OV-Fact: Open-Vocabulary Factuality Evaluation

• OV-Fact is a research approach that decomposes long texts and image captions into atomic facts or candidate entities for precise, verifiable evaluation.
• It employs retrieval-based verification for text and open-vocabulary grounding for visual inputs, ensuring both factual precision and descriptive recall.
• The method underpins metric design for benchmarking and data filtering, demonstrating robust human alignment and improved model performance.

OV-Fact denotes a research direction in factuality evaluation that treats generated content as a collection of verifiable units rather than as a single holistic output. In the current literature, the label appears in two closely related forms: as a general factuality-evaluation style for long-form text generation in "OpenFActScore: Open-Source Atomic Evaluation of Factuality in Text Generation" and as "OVFact," a reference-free metric for long image captions in "OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models" (Lage et al., 8 Jul 2025, Wysoczańska et al., 25 Jul 2025). Both formulations replace coarse document-level judgments with finer-grained verification, but they do so with different primitives: atomic facts in text, and open-vocabulary entities grounded in images.

1. Scope and conceptual basis

OV-Fact rests on the premise that long generations mix correct and incorrect content, so a single correctness label is too coarse for evaluation. In the text setting, this leads to decomposition into atomic facts, defined as minimal information units such that a single sentence may contain multiple factual claims. In the caption setting, the analogous unit is the candidate entity, typically an object phrase with visual attributes such as color or material (Lage et al., 8 Jul 2025, Wysoczańska et al., 25 Jul 2025).

Taken together, these works distinguish two complementary desiderata. The first is factual precision: among the claims or entities that a model states, how many are supported by external evidence. The second is descriptiveness, operationalized in OVFact as recall over visually present entities. This makes OV-Fact broader than hallucination counting alone, because it penalizes both unsupported assertions and under-described outputs (Lage et al., 8 Jul 2025, Wysoczańska et al., 25 Jul 2025).

Dimension	OpenFActScore	OVFact
Verification unit	Atomic facts	Candidate entities
Evidence source	Retrieved Wikipedia passages	Open-vocabulary detection and segmentation
Primary score	Precision over supported claims	F1 over factual precision and descriptiveness

This comparison suggests that OV-Fact is best understood not as a single metric family with one fixed implementation, but as a methodological stance: factuality should be measured through explicit, externally checkable units, with the verification mechanism adapted to the modality.

2. Atomic factuality for long-form text

In the text domain, OpenFActScore is an open-source reimplementation and extension of FActScore for long-form outputs such as biographies and summaries. Its central question is: what fraction of the individual claims in a generation are supported by a trusted knowledge source? The pipeline has two stages. Atomic Fact Generation (AFG) decomposes each sentence into a list of atomic facts, and Atomic Fact Validation (AFV) labels each fact as Supported or Not-Supported using retrieved evidence from Wikipedia (Lage et al., 8 Jul 2025).

Formally, if a generation is decomposed into atomic facts $\mathcal{F} = \{f_1, f_2, \dots, f_N\}$ and the validator assigns $y_i \in \{S, NS\}$, then

$$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$

This is explicitly a precision-based measure. Contradicted and unverifiable claims are both collapsed into $NS$, and recall over all possible true facts is not measured (Lage et al., 8 Jul 2025).

OpenFActScore preserves the original two-stage structure while removing dependence on proprietary systems. It allows any Hugging Face-compatible model to be used for both AFG and AFV, reuses the original sentence splitting, BM25 demonstration retrieval, and GTR-based Wikipedia retrieval, and introduces system prompts and chat-style prompting. In reported experiments, Gemma achieved the best overall performance, and the final open setup obtained a Pearson correlation of approximately $0.99$ with the original FActScore experiments, while preserving model ranking (Lage et al., 8 Jul 2025).

The empirical rationale for this formulation is that claim-level evaluation avoids subjective partially supported labels and yields fine-grained diagnostics. The paper also emphasizes several assumptions: whether an atomic fact is supported should be undebatable, all atomic facts are equally weighted, and the knowledge base should have no relevant internal contradictions or overlaps (Lage et al., 8 Jul 2025).

3. Open-vocabulary factuality for long captions

In the visual-caption domain, OVFact addresses a different failure mode: long captions are diverse, paragraph-length, and often lack exhaustive human references. Existing hallucination metrics are described as short-caption oriented, precision only, closed vocabulary, and strongly dependent on human-annotated references. OVFact therefore defines a reference-free, open-vocabulary pipeline for evaluating whether a long caption is both visually supported and sufficiently descriptive (Wysoczańska et al., 25 Jul 2025).

Given an image–caption pair $(x, y)$, OVFact first parses the caption into candidate entities $\mathcal{C}$ using Gemma2-27B. The prompt asks for visually concrete objects and attributes while excluding non-visual concepts such as sound or atmosphere. It then grounds each candidate with two open-vocabulary tools: OWL-ViTv2 for object detection and OpenSeg for segmentation. The grounded set is

$\mathcal{G} = \mathcal{G}_D \cup \mathcal{G}_S.$

Precision is defined as

$$\text{OVFact}_{\text{Prec}} = \frac{|\mathcal{G}|}{|\mathcal{C}|}.$$

Here, each candidate entity is treated as a visually testable claim; ungrounded entities are effectively counted as hallucinations or unsupported mentions (Wysoczańska et al., 25 Jul 2025).

Recall is constructed differently. When human captions are available, OVFact parses them into a reference set $\mathcal{R}$. In the reference-free regime, it derives $y_i \in \{S, NS\}$0 from the image itself by grounding a concept vocabulary $y_i \in \{S, NS\}$1 of $y_i \in \{S, NS\}$2 unique concepts, formed from Visual Genome, LVIS, Open Images, and Objects365. Candidate and reference entities are embedded with the SigLIP-So400m/14 text encoder, cosine similarities are computed, and recall is the average best-match similarity from each reference entity to the caption’s candidates:

$y_i \in \{S, NS\}$3

The final scalar score is the harmonic mean

$y_i \in \{S, NS\}$4

This design explicitly balances hallucination avoidance and descriptive coverage (Wysoczańska et al., 25 Jul 2025).

The paper reports that LLM-based parsing reaches $y_i \in \{S, NS\}$5 specificity on both DOCCI and Localized Narratives, while grounding specificity reaches $y_i \in \{S, NS\}$6 on DOCCI and $y_i \in \{S, NS\}$7 on Localized Narratives when both detection and segmentation are used. Using only one of the two grounding tools reduces specificity substantially, which is presented as evidence that “stuff-like” regions and object-like regions must both be handled in long-caption evaluation (Wysoczańska et al., 25 Jul 2025).

4. Verification mechanisms and metric design

The two OV-Fact formulations differ chiefly in what counts as evidence. OpenFActScore is retrieval-augmented: each atomic fact is paired with its topic and the top five passages retrieved from Wikipedia using GTR, and an evaluator model must answer “True” or “False.” The final label is determined by parsing the first occurrence of either token. In the reported configuration, output parsing rather than logits-based thresholding is used (Lage et al., 8 Jul 2025).

OVFact is tool-based rather than corpus-retrieval based. It composes an LLM parser, open-vocabulary detectors and segmenters, and a text embedding model. Its “reference-free” character does not mean absence of external evidence; rather, it means that no human-annotated captions or bounding boxes are required for the metric to be defined. The image itself, together with open-vocabulary grounding tools, supplies the evidentiary basis (Wysoczańska et al., 25 Jul 2025).

A central methodological difference follows from this. OpenFActScore is a pure precision metric by construction. It is designed for settings where the key question is whether stated claims are supported by a trusted corpus. OVFact, by contrast, is explicitly designed to avoid rewarding excessively conservative captions. The paper argues that precision-only metrics can make a caption such as “A person outdoors” appear safe even when it misses most of the scene, and therefore combines grounded precision with an embedding-based recall term (Wysoczańska et al., 25 Jul 2025).

This contrast also clarifies a common misconception. Factuality evaluation is not necessarily synonymous with contradiction detection or hallucination rate. In the text setting, factuality can be operationalized as precision over atomic claims. In the long-caption setting, the same operationalization is incomplete unless descriptive recall is also accounted for. The two papers therefore instantiate different points on the precision–recall design space rather than competing definitions of the same quantity (Lage et al., 8 Jul 2025, Wysoczańska et al., 25 Jul 2025).

5. Empirical performance and human alignment

OpenFActScore evaluates four open models for AFG and AFV: Llama 3.1-Instruct, Gemma, Qwen, and Olmo. For AFG, Gemma achieves the best average BERTScore-F1 at approximately $y_i \in \{S, NS\}$8, with Olmo very close at approximately $y_i \in \{S, NS\}$9, Llama 3.1 at approximately $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$0, and Qwen at approximately $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$1. For AFV, cumulative Error Rate relative to human FActScore is $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$2 for Gemma, $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$3 for Llama 3.1, $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$4 for Qwen, and $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$5 for Olmo. Gemma is therefore selected as the default AFV evaluator, while Olmo is selected as the default AFG model because it is fully open and nearly matches Gemma on AFG (Lage et al., 8 Jul 2025).

At the end-to-end level, the open configuration uses Olmo for AFG and Gemma for AFV. Absolute FActScores are lower than in the original closed-source setups, but the ranking of the evaluated LLMs is identical, and Pearson correlation with the original FActScore is greater than $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$6 for both comparison settings. This makes OpenFActScore suitable as an open replacement for benchmarking and ranking, even when score calibration differs in absolute value (Lage et al., 8 Jul 2025).

OVFact evaluates metric quality through side-by-side human preferences on precision and descriptiveness. On these pairwise judgments, OVFact precision agrees with human factuality preferences in $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$7 of cases, compared with $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$8 for ALOHa and $$\text{FActScore} = \frac{\sum_{i=1}^{N} \mathbf{1}[y_i = S]}{N}.$$9 for CLIP-Image-Score. On descriptiveness, OVFact recall agrees with humans in $NS$0 of cases, compared with $NS$1 for CLIP-Image-Score. These results are notable because OVFact achieves this alignment without access to ground-truth captions, whereas ALOHa depends on them (Wysoczańska et al., 25 Jul 2025).

The reported interpretation is that open-vocabulary grounding and max-based recall over reference entities behave more robustly than Hungarian matching when references are incomplete. This is especially important in long-caption settings, where captions are diverse and human reference sets are often missing or non-exhaustive (Wysoczańska et al., 25 Jul 2025).

6. Data filtering, limitations, and broader significance

OVFact is not only an evaluation metric but also a data-selection mechanism. The paper ranks image–caption pairs in ShareGPT4V by $NS$2 and trains PaliGemma 2 on filtered subsets that are $NS$3–$NS$4 smaller than the original dataset. Across DOCCI, Localized Narratives, and COCO CHAIR evaluations, models trained on OVFact-filtered data show improved factual precision without sacrificing caption descriptiveness. In a direct side-by-side human comparison between a model trained on full ShareGPT4V and a model trained on the top $NS$5 of ShareGPT4V ranked by OVFact, annotators prefer the filtered-data model’s outputs in $NS$6 of comparisons (Wysoczańska et al., 25 Jul 2025).

OpenFActScore points to a parallel but somewhat different downstream use: because it is open, local, and modular, it is well suited for offline large-scale evaluation, benchmarking, model selection, prompt comparison, and even RLHF or RL-style tuning using FActScore or atomic-fact labels as a reward signal. The paper explicitly notes that the pipeline can be adapted to other knowledge sources beyond Wikipedia, which suggests a route toward domain-specific factuality evaluation (Lage et al., 8 Jul 2025).

Both approaches are nevertheless bounded by the quality of their evidence interfaces. OpenFActScore depends on the coverage and retrieval quality of Wikipedia and GTR, collapses contradictions and unverifiable claims into a single Not-Supported category, and weights all atomic facts equally. OVFact inherits biases and failures from Gemma2-27B, OWL-ViTv2, OpenSeg, and SigLIP; it focuses on object-level factuality rather than complex relational structure, and its scaling to very large datasets remains computationally nontrivial (Lage et al., 8 Jul 2025, Wysoczańska et al., 25 Jul 2025).

The broader significance of OV-Fact lies in this shift from reference overlap to explicit verification. In text generation, this yields atomic, retrieval-grounded precision over claims. In long-caption modeling, it yields open-vocabulary, reference-free factuality with an explicit descriptiveness term. This suggests a unifying research program in which factuality is evaluated through externally grounded, fine-grained units, while the choice of units, evidence source, and aggregation rule is made modality-specific rather than assumed universal (Lage et al., 8 Jul 2025, Wysoczańska et al., 25 Jul 2025).