FinReflectKG – EvalBench

• FinReflectKG – EvalBench is a benchmark framework designed to standardize evaluation of financial knowledge graph extraction from SEC 10-K filings.
• It assesses candidate triples along four dimensions—faithfulness, precision, relevance, and comprehensiveness—using a deterministic commit-then-justify LLM-as-Judge protocol.
• The framework compares single-pass, multi-pass, and reflection-agent-based extraction modes to balance conservative factual grounding against comprehensive fact coverage.

Searching arXiv for the named benchmark and closely related FinReflectKG papers to ground the article in current sources. to=arxiv_search.query 久久免费热在线精品{"^^^^^^^^4search_query4 - EvalBench\" OR ^^^^^^^^4all:\4 EvalBench\" OR ^^^^^^^^4all:\4 to=arxiv_search.query 天天中彩票人工json content='&&&^^^^4search_query4^^^^&&&)","title":"FinReflectKG -- HalluBench: GraphRAG Hallucination Benchmark for Financial Question Answering Systems","authors":["Wei Qin","Minghan Xie","Tianyang Wang","Yaqing Wang","Zixiang Wang","Weijie Wang","Jian Gao","Minghong Sun"],"abstract":"As organizations increasingly integrate AI-powered question-answering systems into financial information systems for compliance, risk assessment, and decision support, ensuring the factual accuracy of AI-generated outputs becomes a critical engineering challenge. Current Knowledge Graph (KG)-augmented QA systems lack systematic mechanisms to detect hallucinations - factually incorrect outputs that undermine reliability and user trust. We introduce FinBench-QA-Hallucination, a benchmark for evaluating hallucination detection methods in KG-augmented financial QA over SEC ^^^^4all:\4search_query4^^^^-K filings. The dataset contains 755 annotated examples from ^^^^4 OR all:\4search_query4search_query4^^^^ pages, each labeled for groundedness using a conservative evidence-linkage protocol requiring support from both textual chunks and extracted relational triplets. We evaluate six detection approaches - LLM judges, fine-tuned classifiers, Natural Language Inference (NLI) models, span detectors, and embedding-based methods under two conditions: with and without KG triplets. Results show that LLM-based judges and embedding approaches achieve the highest performance (F^^^^4all:\4^^^^: ^^^^4search_query4^^^^.8^^^^4 OR all:\4^^^^-^^^^4search_query4^^^^.86) under clean conditions. However, most methods degrade significantly when noisy triplets are introduced, with Matthews Correlation Coefficient (MCC) dropping 44-84 percent, while embedding methods remain relatively robust with only 9 percent degradation. Statistical tests (Cochran's Q and McNemar) confirm significant performance differences (p < ^^^^4search_query4^^^^.^^^^4search_query4search_query4all:\4 Our findings highlight vulnerabilities in current KG-augmented systems and provide insights for building reliable financial information systems, where hallucinations can lead to regulatory violations and flawed decisions. The benchmark also offers a framework for integrating AI reliability evaluation into information system design across other high-stakes domains such as healthcare, legal, and government.","categories":["cs.IR","cs.AI"]},{"arxiv_id":"(&&&^^^^4all:\4^^^^&&&)","title":"FinReflectKG - EvalBench: Benchmarking Financial KG with Multi-Dimensional Evaluation","authors":["Wei Qin","Minghan Xie","Tianyang Wang","Yaqing Wang","Zixiang Wang","Weijie Wang","Jian Gao","Minghong Sun"],"abstract":"LLMs are increasingly being used to extract structured knowledge from unstructured financial text. Although prior studies have explored various extraction methods, there is no universal benchmark or unified evaluation framework for the construction of financial knowledge graphs (KG). We introduce FinReflectKG - EvalBench, a benchmark and evaluation framework for KG extraction from SEC ^^^^4all:\4search_query4^^^^-K filings. Building on the agentic and holistic evaluation principles of FinReflectKG - a financial KG linking audited triples to source chunks from S&P ^^^^4all:\4search_query4search_query4^^^^ filings and supporting single-pass, multi-pass, and reflection-agent-based extraction modes - EvalBench implements a deterministic commit-then-justify judging protocol with explicit bias controls, mitigating position effects, leniency, verbosity and world-knowledge reliance. Each candidate triple is evaluated with binary judgments of faithfulness, precision, and relevance, while comprehensiveness is assessed on a three-level ordinal scale (good, partial, bad) at the chunk level. Our findings suggest that, when equipped with explicit bias controls, LLM-as-Judge protocols provide a reliable and cost-efficient alternative to human annotation, while also enabling structured error analysis. Reflection-based extraction emerges as the superior approach, achieving best performance in comprehensiveness, precision, and relevance, while single-pass extraction maintains the highest faithfulness. By aggregating these complementary dimensions, FinReflectKG - EvalBench enables fine-grained benchmarking and bias-aware evaluation, advancing transparency and governance in financial AI applications.","categories":["cs.AI","cs.CL","q-fin.GN"]},{"arxiv_id":"(&&&^^^^4 OR all:\4^^^^&&&)","title":"FinReflectKG - MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence","authors":["Wei Qin","Minghan Xie","Tianyang Wang","Yaqing Wang","Zixiang Wang","Weijie Wang","Jian Gao","Minghong Sun"],"abstract":"Multi-hop reasoning over financial disclosures is often a retrieval problem before it becomes a reasoning or generation problem: relevant facts are dispersed across sections, filings, companies, and years, and LLMs often expend excessive tokens navigating noisy context. Without precise Knowledge Graph (KG)-guided selection of relevant context, even strong reasoning models either fail to answer or consume excessive tokens, whereas KG-linked evidence enables models to focus their reasoning on composing already retrieved facts. We present FinReflectKG - MultiHop, a benchmark built on FinReflectKG, a temporally indexed financial KG that links audited triples to source chunks from S&P ^^^^4all:\4search_query4search_query4^^^^ filings (^^^^4 OR all:\4search_query4 OR all:\4 OR all:\4^^^^-^^^^4 OR all:\4search_query4 OR all:\44^^^^). Mining frequent ^^^^4 OR all:\4^^^^-^^^^4 OR all:\4^^^^ hop subgraph patterns across sectors (via GICS taxonomy), we generate financial analyst style questions with exact supporting evidence from the KG. A two-phase pipeline first creates QA pairs via pattern-specific prompts, followed by a multi-criteria quality control evaluation to ensure QA validity. We then evaluate three controlled retrieval scenarios: (S^^^^4all:\4^^^^) precise KG-linked paths; (S^^^^4 OR all:\4^^^^) text-only page windows centered on relevant text spans; and (S^^^^4 OR all:\4^^^^) relevant page windows with randomizations and distractors. Across both reasoning and non-reasoning models, KG-guided precise retrieval yields substantial gains on the FinReflectKG - MultiHop QA benchmark dataset, boosting correctness scores by approximately ^^^^4 OR all:\44^^^^ percent while reducing token utilization by approximately 84.5 percent compared to the page window setting, which reflects the traditional vector retrieval paradigm. Spanning intra-document, inter-year, and cross-company scopes, our work underscores the pivotal role of knowledge graphs in efficiently connecting evidence for multi-hop financial QA. We also release a curated subset of the benchmark (555 QA Pairs) to catalyze further research.","categories":["cs.CL","cs.AI","q-fin.GN"]},{"arxiv_id":"(&&&^^^^4 OR all:\4^^^^&&&)","title":"FinReflectKG: Agentic Construction and Evaluation of Financial Knowledge Graphs","authors":["Wei Qin","Minghan Xie","Tianyang Wang","Yaqing Wang","Zixiang Wang","Weijie Wang","Jian Gao","Minghong Sun"],"abstract":"The financial domain poses unique challenges for knowledge graph (KG) construction at scale due to the complexity and regulatory nature of financial documents. Despite the critical importance of structured financial knowledge, the field lacks large-scale, open-source datasets capturing rich semantic relationships from corporate disclosures. We introduce an open-source, large-scale financial knowledge graph dataset built from the latest annual SEC ^^^^4all:\4search_query4^^^^-K filings of all S and P ^^^^4all:\4search_query4search_query4^^^^ companies - a comprehensive resource designed to catalyze research in financial AI. We propose a robust and generalizable knowledge graph (KG) construction framework that integrates intelligent document parsing, table-aware chunking, and schema-guided iterative extraction with a reflection-driven feedback loop. Our system incorporates a comprehensive evaluation pipeline, combining rule-based checks, statistical validation, and LLM-as-a-Judge assessments to holistically measure extraction quality. We support three extraction modes - single-pass, multi-pass, and reflection-agent-based - allowing flexible trade-offs between efficiency, accuracy, and reliability based on user requirements. Empirical evaluations demonstrate that the reflection-agent-based mode consistently achieves the best balance, attaining a 64.8 percent compliance score against all rule-based policies (CheckRules) and outperforming baseline methods (single-pass and multi-pass) across key metrics such as precision, comprehensiveness, and relevance in LLM-guided evaluations.","categories":["cs.AI","cs.CL","q-fin.GN"]}]'【อ่านข้อความเต็มanalysis to=arxiv_search.query аҵք content=''} FinReflectKG – EvalBench is a benchmark and evaluation framework for financial knowledge-graph extraction from SEC ^^^^4all:\4search_query4^^^^-K filings, introduced by Wei Qin, Minghan Xie, Tianyang Wang, Yaqing Wang, Zixiang Wang, Weijie Wang, Jian Gao, and Minghong Sun (&&&^^^^4all:\4^^^^&&&). It was designed to address the absence of a universal benchmark or unified evaluation framework for the construction of financial knowledge graphs from long, heterogeneous, and high-stakes disclosures. Rather than reducing extraction quality to a single scalar score, it evaluates candidate triples along four dimensions—faithfulness, precision, relevance, and comprehensiveness—using a deterministic commit-then-justify LLM-as-Judge protocol with explicit controls for leniency, position effects, verbosity bias, and world-knowledge reliance (&&&^^^^4all:\4^^^^&&&).

^^^^4all:\4^^^^. Position within the FinReflectKG framework

EvalBench is positioned as the evaluation layer of the broader FinReflectKG system. FinReflectKG is described as an open-source, large-scale financial KG dataset and construction framework built from the latest annual SEC ^^^^4all:\4search_query4^^^^-K filings of all S&P ^^^^4all:\4search_query4search_query4^^^^ companies for ^^^^4 OR all:\4search_query4 OR all:\44^^^^, with intelligent document parsing, table-aware chunking, schema-guided iterative extraction, and a reflection-driven feedback loop (&&&^^^^4 OR all:\4^^^^&&&). Within that larger setting, EvalBench operationalizes how extracted financial triples should be judged once they have been linked back to source chunks.

The benchmark inherits several defining properties from FinReflectKG. It is built around audited triples linked to source chunks, it assumes a closed-schema extraction setting rather than unconstrained open extraction, and it compares three extraction paradigms: single-pass, multi-pass, and reflection-agent-based extraction (&&&^^^^4all:\4^^^^&&&). This makes EvalBench not merely a leaderboard for a single model, but a comparative framework for studying how different extraction regimes trade off conservatism, structural normalization, and iterative correction.

A common misconception is that FinReflectKG – EvalBench is a generic financial QA benchmark. It is more precisely a benchmark for financial KG extraction. Its downstream significance for QA and GraphRAG is substantial, but its immediate object of evaluation is the quality of triples extracted from filing text, not answer generation itself (&&&^^^^4all:\4^^^^&&&).

^^^^4 OR all:\4^^^^. Extraction setting and source corpus

The benchmark is built on U.S. SEC Form ^^^^4all:\4search_query4^^^^-K filings from the S&P ^^^^4all:\4search_query4search_query4^^^^ companies for fiscal year ^^^^4 OR all:\4search_query4 OR all:\44^^^^ (&&&^^^^4all:\4^^^^&&&). The extraction problem is formalized over a corpus PRESERVED_PLACEHOLDER_^^^^4search_query4^^^^ of SEC ^^^^4all:\4search_query4^^^^-K documents, where each document PRESERVED_PLACEHOLDER_^^^^4all:\4^^^^ is segmented into deterministic, structure-aware text chunks,

PRESERVED_PLACEHOLDER_^^^^4 OR all:\4^^^^

For each chunk PRESERVED_PLACEHOLDER_^^^^4 OR all:\4^^^^, an extractor $E$ produces a set of candidate triples,

$$T_x = E(x) \subseteq \mathcal{T}, \qquad t = (s, r, o) \in \mathcal{T}.$$

This chunk-linked formulation is central to the benchmark because every judgment is localized to the source span from which the triple was generated (&&&^^^^4all:\4^^^^&&&).

EvalBench compares three extraction modes. In single-pass extraction, a single LLM performs extraction and normalization in one shot. In multi-pass extraction, one LLM extracts candidate triples and another normalizes them according to rubric parameters. In reflection-agent-based extraction, extraction and reflection are repeated until inconsistencies are resolved or a maximum iteration limit is reached (&&&^^^^4all:\4^^^^&&&). The benchmark is therefore as much about extraction strategy as about extraction accuracy.

This architecture reflects the broader FinReflectKG design philosophy. The upstream framework emphasizes agentic and holistic construction, while EvalBench supplies a correspondingly structured assessment layer. A plausible implication is that the benchmark is intended to support design-space exploration: not only whether a system extracts facts, but how its extraction protocol affects different quality dimensions.

^^^^4 OR all:\4^^^^. Evaluation dimensions and scoring semantics

EvalBench uses four complementary evaluation dimensions (&&&^^^^4all:\4^^^^&&&).

Faithfulness measures whether a triple is grounded in the source text and does not rely on world knowledge or bridging inference. A triple can be plausible in financial discourse and still fail faithfulness if it is not directly supported by the chunk.

Precision measures whether the triple is specific and clear, penalizing generic placeholders such as “Company,” vague relations, and inaccurate quantities or dates. Precision is therefore not synonymous with faithfulness: a triple may be supported yet imprecise.

Relevance checks whether the triple is directly related to the main theme of the source span rather than introducing tangential content. This distinguishes central disclosures from incidental details.

Comprehensiveness measures coverage of the chunk’s atomic facts. Unlike the other three dimensions, it is assessed at the chunk level rather than the individual triple level, using a three-level ordinal scale: good, partial, and bad (&&&^^^^4all:\4^^^^&&&).

The aggregation logic is also explicitly asymmetric. Faithfulness, precision, and relevance are evaluated per triple and micro-averaged across all extracted triples, whereas comprehensiveness is evaluated per chunk and macro-averaged across chunks (&&&^^^^4all:\4^^^^&&&). This is a consequential design choice: local correctness is measured at the triple level, but informational coverage is treated as a chunk-level property.

The appendix examples clarify the intended semantics. A faithful triple must have direct textual support; a precise triple must use exact entities, relation semantics, and numeric details; a relevant triple must encode a core fact rather than an incidental modifier; and a comprehensive set must cover the core facts of the chunk rather than a selective subset (&&&^^^^4all:\4^^^^&&&). Taken together, these dimensions make EvalBench a genuinely multi-objective benchmark rather than a proxy for one latent notion of “quality.”

4. Deterministic commit-then-justify judging

A major methodological contribution of EvalBench is its deterministic commit-then-justify LLM-as-Judge protocol (&&&^^^^4all:\4^^^^&&&). The judge is instantiated with Qwen^^^^4 OR all:\4^^^^-^^^^4 OR all:\4 OR all:\4^^^^B, and decoding is deterministic with temperature = ^^^^4search_query4^^^^.^^^^4search_query4. The protocol requires the model first to produce a structured verdict and only then a brief justification of up to ^^^^4all:\4^^^^5 words. The benchmark also introduces a warning signal that flags likely error types and suggests an actionable correction path.

The required output format is a JSON array only:

{"verdict":0|1,"reasoning":"...","warning":"..."}

This structure is intended to stabilize judge behavior and support downstream error analysis. Warnings can indicate phenomena such as hallucination, generic entities, amount mismatch, off-topic extraction, missing information, and possible positional bias (&&&^^^^4all:\4^^^^&&&).

EvalBench is explicit that LLM judges are vulnerable to systematic bias, and it introduces prompt-level controls against several well-known failure modes. Leniency bias is countered by a conservative rule: if evidence is ambiguous, return ^^^^4search_query4^^^^. World-knowledge reliance is suppressed by a locality constraint forbidding external information or inference beyond the source text. Position effects are mitigated by instructing the judge not to let sentence position or placement affect the verdict. Verbosity bias is reduced by instructing the judge not to let length or surface form affect the decision. Few-shot examples for each criterion are included for calibration and consistency (&&&^^^^4all:\4^^^^&&&).

These controls are important because a second misconception about EvalBench is that it simply substitutes an unconstrained LLM for human annotation. The benchmark’s central claim is narrower and more technical: with deterministic decoding, explicit locality, conservative scoring, and anti-bias instructions, LLM-as-Judge can function as a reliable and cost-efficient alternative to human annotation while enabling structured error analysis (&&&^^^^4all:\4^^^^&&&).

5. Empirical results and extraction trade-offs

The reported results exhibit a clear pattern. Reflection is best on comprehensiveness, precision, and relevance, while single-pass is best on faithfulness (&&&^^^^4all:\4^^^^&&&).

For Single Pass, the reported scores are: comprehensiveness 6^^^^4 OR all:\4^^^^.6^^^^4search_query4^^^^, faithfulness 87.^^^^4 OR all:\4^^^^5, precision 56.^^^^4search_query4^^^^6, and relevance 9^^^^4all:\4^^^^.46. For Multi Pass, they are: comprehensiveness 6^^^^4 OR all:\4^^^^.4^^^^4all:\4^^^^, faithfulness 78.7^^^^4 OR all:\4^^^^, precision 58.^^^^4search_query4all:\4^^^^, and relevance 8^^^^4 OR all:\4^^^^.64. For Reflection, they are: comprehensiveness 7^^^^4 OR all:\4^^^^.^^^^4search_query4all:\4^^^^, faithfulness 8^^^^4 OR all:\4^^^^.4^^^^4search_query4^^^^, precision 59.49, and relevance 9^^^^4 OR all:\4^^^^.5^^^^4 OR all:\4^^^^ (&&&^^^^4all:\4^^^^&&&).

The comparative interpretation given in the paper is that reflection improves coverage and often structural quality because iterative refinement finds more atomic facts, but larger coverage also increases the chance of drifting beyond the source text, which can reduce strict faithfulness (&&&^^^^4all:\4^^^^&&&). Single-pass extraction is more conservative and therefore more faithful, but it misses more facts. Multi-pass extraction improves structural consistency relative to single-pass, yet does not dominate the reflection mode on the broader multi-dimensional profile.

This trade-off is methodologically significant. EvalBench does not present “best extraction” as a one-dimensional concept. Instead, it exposes a Pareto structure: conservatism favors faithfulness, while iterative correction favors coverage and broader semantic utility. This suggests that extraction-mode choice should depend on downstream requirements—for example, whether a deployment prioritizes provable source grounding or richer graph coverage.

6. Role in the broader FinReflectKG benchmark family

EvalBench is most naturally understood as the upstream evaluation substrate in a wider FinReflectKG benchmark family. Two later benchmarks make this continuity visible.

“FinReflectKG - MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence” evaluates financial multi-hop QA under controlled evidence regimes: S^^^^4all:\4^^^^ precise KG-linked paths, S^^^^4 OR all:\4^^^^ text-only page windows centered on relevant text spans, and S^^^^4 OR all:\4^^^^ relevant page windows with randomizations and distractors. It reports that KG-guided precise retrieval boosts correctness scores by approximately ^^^^4 OR all:\44^^^^ percent while reducing token utilization by approximately 84.5 percent compared to the page-window setting (&&&^^^^4 OR all:\4^^^^&&&). The benchmark relies on a temporally indexed financial KG linking audited triples to source chunks, which is consistent with the extraction-and-provenance assumptions established in EvalBench.

“FinReflectKG -- HalluBench: GraphRAG Hallucination Benchmark for Financial Question Answering Systems,” also referred to in the paper as FinBench-QA-Hallucination, evaluates hallucination detection in KG-augmented financial QA over SEC ^^^^4all:\4search_query4^^^^-K filings. It contains 755 annotated examples from ^^^^4 OR all:\4search_query4search_query4^^^^ pages, labels groundedness using a conservative evidence-linkage protocol requiring support from both textual chunks and extracted relational triplets, and compares detection methods with and without KG triplets (&&&^^^^4 OR all:\44^^^^&&&). Its results show that LLM-based judges and embedding approaches achieve the highest performance under clean conditions, but most methods degrade significantly when noisy triplets are introduced (&&&^^^^4 OR all:\44^^^^&&&).

This suggests that EvalBench serves a foundational role: it evaluates whether the KG extraction layer is faithful, precise, relevant, and comprehensive before that graph is used for retrieval, multi-hop reasoning, or hallucination detection. In that sense, FinReflectKG – EvalBench is not only a benchmark for extraction quality; it is also an infrastructure benchmark for downstream financial AI systems whose reliability depends on the quality of graph-grounded evidence.