Factual-Aware Evaluator (FAE)

• Factual-Aware Evaluator (FAE) is a framework that assesses model outputs against grounded evidence rather than relying solely on stylistic cues.
• It employs fine-grained decomposition and hierarchical scoring to deliver localized veracity checks and diagnose factual inconsistencies.
• FAE systems integrate domain-specific grounding mechanisms and perturbation-based testing to enhance the reliability of factual evaluation.

Searching arXiv for papers relevant to “Factual-Aware Evaluator (FAE)” and closely related factuality-evaluation frameworks. A factual-aware evaluator (FAE) is best understood as an evaluator that judges model outputs against grounding signals rather than stylistic plausibility alone. In current work, that role appears in several forms: a knowledge-graph fact checker that assigns a truthfulness score to a triple using evidence paths (Pirrò, 2020), a causality-inspired evaluator of chain-of-thought faithfulness and answer support (Sun et al., 12 Apr 2026), a claim-level critic grounded in retrieved documents (Xie et al., 2024), a reference-free multimodal judge that scores captions against video content (Dipta et al., 20 Sep 2025), a factual rubric-based evaluator for text-to-image outputs (Lim et al., 18 May 2026), and, in Video-PLR, an explicit “Factual-Aware Evaluator” that scores whether a timestamped segment description is consistent with the corresponding video clip (Pu et al., 23 Nov 2025). Taken together, these systems define FAE less as a single architecture than as a family of evaluators whose central object is grounded correctness.

1. Scope and defining characteristics

In knowledge-graph fact checking, the evaluation target is a triple $(s,p,o)$, and the system predicts a truthfulness score $\Phi_s \in [0,1]$ from paths connecting subject and object in the graph (Pirrò, 2020). In reasoning evaluation, the target is a chain $\mathbf{S}$, and the evaluator scores both whether the chain supports the correct answer and whether its intermediate steps are faithful to one another (Sun et al., 12 Apr 2026). In document-grounded text evaluation, the target is a claim $c$ against evidence $d$, with outputs such as Supported, Contradictory, or Unverified (Xie et al., 2024). In video caption evaluation, the target is a pair $(V,\tilde{X})$, and the model returns a score $S \in \{1,\dots,5\}$ plus an explanation (Dipta et al., 20 Sep 2025). In text-to-image factuality, the target is whether an image satisfies visually verifiable facts grounded in or implied by the prompt, using a structured factual rubric and question-answer checks (Lim et al., 18 May 2026). In video reasoning, the evaluator receives $\text{clip}(Video,t_{start},t_{end})$ and a segment description $desc$, then outputs a probability that the description is correct (Pu et al., 23 Nov 2025).

These formulations share a common pattern. The evaluator is not merely an answer checker; it is a grounding module. Depending on domain, the grounding signal may be a TBox/ABox path structure, retrieved documents, reference images, the raw video itself, or controlled perturbations of a reasoning trace. This suggests that FAE is best treated as a general evaluation paradigm for grounded fidelity, with modality-specific realizations.

2. Units of analysis and granularity

A defining property of recent FAE-style systems is fine-grained decomposition. In FEA, evidence is not the triple alone but a set of schema-conditioned data-level paths, with each path represented as a sequence of embedded facts $\pi_E=[t_E^1,t_E^2,\ldots,t_E^l]$ (Pirrò, 2020). In FACT-E, the relevant unit is the split-chain pair $\Phi_s \in [0,1]$0, so faithfulness is analyzed locally at each prefix/suffix boundary rather than only at the final answer (Sun et al., 12 Apr 2026). In FenCE, the elementary judgment is $\Phi_s \in [0,1]$1, with $\Phi_s \in [0,1]$2, and response-level scoring is obtained by computing the percentage of factual claims (Xie et al., 2024). Facts&Evidence decomposes a passage into sentences, then into atomic claims, then into claim–evidence judgments, and finally back into sentence- and document-level credibility (Boonsanong et al., 19 Mar 2025). FELM adopts segment-level annotation because it is closest to a user-facing output format, while also noting that claim-based evaluation can be effective as an internal intermediate representation (Chen et al., 2023). FaaF constructs a fact list $\Phi_s \in [0,1]$3 and verifies all facts associated with one generated answer in a single function call (Katranidis et al., 2024).

The central design choice is therefore not only what to score, but at what granularity to expose the score. Sentence-level and segment-level outputs support localization; claim-level outputs support diagnosis; path-level and step-level outputs support structural explanation. A plausible implication is that strong FAEs are usually hierarchical: they compute local grounded judgments first and aggregate later.

3. Grounding signals and verification mechanisms

The grounding mechanism varies by domain, but recent systems consistently move away from unconstrained self-judgment. FEA uses a schema-first pipeline: it builds a TBox graph, computes predicate relatedness $\Phi_s \in [0,1]$4, extracts top-$\Phi_s \in [0,1]$5 schema-level paths, instantiates them as ABox paths, embeds each fact, aggregates paths, and outputs $\Phi_s \in [0,1]$6 (Pirrò, 2020). FenCE grounds each claim in heterogeneous external evidence from Bing Web Search API, an offline Wikipedia dump, and Google Knowledge Graph API, reranking candidate documents with GTR-T5-Large and feeding the top 5 documents to the evaluator (Xie et al., 2024). Facts&Evidence performs Google web search through Serper, scrapes top websites, extracts readable text with Mozilla Readability, retrieves relevant passages with jina-embeddings-v3, tags source types such as news, blog, wiki, social media, scientific/medical article, and government website, and then judges support claim by claim (Boonsanong et al., 19 Mar 2025). FACTOR uses hybrid retrieval over Wikipedia by combining BM25 and dense retrieval, then applies a DeBERTa-based NLI cross-encoder with stricter verification thresholds for higher-uncertainty claims (Hassan et al., 21 Jun 2026).

In multimodal settings, the evidence channel becomes the input signal itself. VC-Inspector is reference-free and judges a caption directly against the video, with synthetic supervision created by object and action substitutions (Dipta et al., 20 Sep 2025). FAGER combines LLM-based fact proposal with reference-guided visual fact extraction and verification, converts the resulting factual rubric into atomic visual QA pairs, and evaluates a generated image via yes/no/unknown answers grounded strictly in visible evidence (Lim et al., 18 May 2026). Video-PLR’s FAE evaluates text-video consistency for each timestamped clip-description pair and uses the resulting score as an anti-hallucination reward (Pu et al., 23 Nov 2025).

A separate line replaces evidence retrieval with controlled interventions. FACT-E introduces external perturbation noise $\Phi_s \in [0,1]$7 and estimates intra-chain faithfulness by comparing original and perturbed continuations across split points (Sun et al., 12 Apr 2026). FBI evaluates factual accuracy, instruction following, coherence, and reasoning by constructing perturbations that should clearly degrade one capability and measuring whether evaluator LLMs notice the drop (Doddapaneni et al., 2024). This suggests two distinct but compatible FAE traditions: evidence-grounded verification and perturbation-based sensitivity testing.

4. Scoring functions and decision rules

Recent FAE-style systems use several recurrent scoring forms. FEA predicts a truthfulness score $\Phi_s \in [0,1]$8, where $\Phi_s \in [0,1]$9 is the set of paths connecting subject and object (Pirrò, 2020). FACT-E defines a reliability score for a chain as

$\mathbf{S}$0

where $\mathbf{S}$1 is intra-chain faithfulness and $\mathbf{S}$2 is CoT-to-answer consistency (Sun et al., 12 Apr 2026). FenCE uses a three-way label space at claim level and then scores a response by the percentage of factual claims (Xie et al., 2024). Facts&Evidence computes a credibility score as

$\mathbf{S}$3

then highlights sentences in $\mathbf{S}$4, $\mathbf{S}$5, and $\mathbf{S}$6 credibility ranges (Boonsanong et al., 19 Mar 2025). FaaF constrains outputs with typed function schemas and enum values such as True, False, and Not clear from the given passage, with the latter optionally mapped to False for binary accounting (Katranidis et al., 2024).

Multimodal systems often expose more human-readable scales. VC-Inspector returns $\mathbf{S}$7, where $\mathbf{S}$8 and $\mathbf{S}$9 is an explanation (Dipta et al., 20 Sep 2025). FAGER assigns yes $c$0, no $c$1, unknown $c$2, averages over factual QA checks, and uses a regeneration threshold of 20 and a keep threshold of 95 in its coarse-to-fine decision process (Lim et al., 18 May 2026). FACTOR computes claim-level uncertainty from token entropy and semantic consistency, then routes claims into Low $c$3, Mid $c$4, and High $c$5 tiers with NLI thresholds $c$6, $c$7, and $c$8, respectively, and requires multiple-passage support for high-uncertainty claims (Hassan et al., 21 Jun 2026). In Video-PLR, FAE outputs $c$9 and $d$0 for each clip-description pair, and the anti-hallucination reward aggregates $d$1 with redundancy attenuation based on temporal IoU and ROUGE overlap (Pu et al., 23 Nov 2025).

These decision rules show that FAE scoring is not uniform. Some systems are binary or ternary judges; some are calibrated scalar scorers; some are rerankers; some are reward models. What remains stable is that the score is grounded in an external structure rather than a free-form quality impression.

5. Representative instantiations across domains

In structured knowledge, FEA combines schema-guided evidence discovery, triple-level path representation, and neural aggregation. Its experiments show that direct triple embeddings via triple2vec often outperform DistMult and ComplEx compositions, that LSTMMaxPool generally performs best among the tested aggregators, and that top-$d$2 predicate-related filtering matters, with $d$3 usually giving the best tradeoff (Pirrò, 2020). In sparse MoE LLMs, expert-aware causal tracing asks whether factual recall localizes at the layer, expert, or coalition level. For Qwen3-30B-A3B-Base, layer 44 and expert L44E069 validate strongly, while for Mixtral-8x7B-v0.1 singleton localization fails and coalition recovery at layer 19 better explains factual signal (Lu et al., 2 Jun 2026). This suggests that factual attribution in MoE settings is model- and protocol-dependent rather than universal.

In reasoning evaluation, FACT-E reports that standard FACT-E is best or second-best in 8/12 model-dataset settings, while lightweight FACT-E is best or second-best in 11/12 such settings, and that $d$4 evaluation trials is a practical sweet spot because gains are strongest from 2 to 3 trials and then saturate or fluctuate (Sun et al., 12 Apr 2026). In long-form generation, FACTOR improves FactScore from $d$5 for Static Constr. to $d$6, reduces hallucination rate from 63.2 to 57.7, and lowers average verification calls from 194.0 to 41.9, although generation latency rises from 22.8s to 51.0s (Hassan et al., 21 Jun 2026). In generator improvement, FenCE raises evaluator accuracy by 2.9 points on LLM-AggreFact over its vanilla SFT version and improves Llama3-8B-chat’s factuality rate on FActScore from 50.96% to 65.41% in its full Edit/Remove + Coarse setting (Xie et al., 2024).

In visual evaluation, VC-Inspector-7B reaches $d$7 and $d$8 on VATEX-Eval in the no-reference setting, outperforming CLIPScore, EMScore, FactVC, PAC-S, ViCLIPScore, and the vanilla Qwen2.5-VL baselines reported there (Dipta et al., 20 Sep 2025). FAGER achieves pairwise factual A/B accuracies of 0.73 on I-HallA-Science, 0.83 on I-HallA-History, 0.82 on ABO, 0.97 on Culture, and 0.87 on T2I-FactualBench-SKCM, exceeding VQAScore and FineGRAIN on all five datasets (Lim et al., 18 May 2026). In video reasoning, Video-PLR’s FAE slightly surpasses GPT-4o on VideoHallucer, with 75.4 accuracy against 74.6, and improves downstream performance when the anti-hallucination reward is combined with the proposed normalization (Pu et al., 23 Nov 2025).

These results do not describe one universal evaluator architecture. They instead show that factual-aware evaluation has become a cross-domain systems pattern: schema-conditioned path checking, claim-evidence critique, rubric-based QA, uncertainty-aware verification, preference-trained multimodal judging, and reward modeling all instantiate the same grounded-evaluation principle.

6. Failure modes, controversies, and limitations

A persistent misconception is that fluent explanation implies faithful evaluation. FACT-E explicitly argues that self-evaluation is confounded by self-affirmation bias and statistical shortcut bias, and that a chain may be linguistically smooth while still being logically invalid (Sun et al., 12 Apr 2026). ChatGPT as a summarization evaluator illustrates related problems: preference for more lexically similar candidates, false reasoning, and inadequate understanding of instructions (Luo et al., 2023). FBI sharpens the critique by showing that evaluator LLMs failed to identify quality drops in over 50% of cases on average, with single-answer and pairwise evaluations showing notable limitations and reference-based evaluations performing comparatively better (Doddapaneni et al., 2024). FELM reaches a similar conclusion from a benchmarking angle: retrieval helps factuality evaluation, but current LLMs are far from satisfactory to faithfully detect factual errors across world knowledge, science and technology, recommendation and writing, reasoning, and math (Chen et al., 2023).

FAE-style systems also inherit domain-specific bottlenecks. Facts&Evidence attributes 53.3% of sampled errors to incorrect evidence retrieval, 26.7% to incorrect atomic claim generation, and 20.0% to incorrect factuality judgment, while taking 30s–1min for passages of 5–7 sentences (Boonsanong et al., 19 Mar 2025). VC-Inspector focuses mainly on factual errors in objects and actions and does not explicitly model attributes, temporal coherence, or narrative structure (Dipta et al., 20 Sep 2025). FAGER depends on multiple LLM/VLM components, reference images, and visible evidence, and it does not report component-level ablations for proposal-only versus reference-only rubric construction (Lim et al., 18 May 2026). FACTOR leaves uncertainty calibration underspecified and trades lower verification cost for higher end-to-end latency (Hassan et al., 21 Jun 2026). In expert-aware MoE tracing, the authors explicitly caution against reading router selection as causal responsibility and against assuming a universal “factual expert” (Lu et al., 2 Jun 2026).

The broader implication is that FAE should not be treated as a solved module. Current work supports several robust design principles—fine-grained decomposition, evidence grounding, perturbation sensitivity, structured outputs, and explicit uncertainty or abstention states—but it also shows that factual evaluation remains limited by retrieval quality, annotation scope, domain shift, evidence incompleteness, and evaluator bias. This suggests that future FAE systems will likely be strongest when they separate internal process faithfulness from external grounding, expose multi-granular evidence traces, and validate themselves with targeted perturbations rather than aggregate correlation alone.