FinReflectKG: Financial Knowledge Graph Framework

• FinReflectKG is a comprehensive financial knowledge graph that extracts structured data from SEC 10-K filings and links each fact to its original evidence.
• It employs intelligent document parsing, table-aware chunking, and a reflection-driven extraction process to enhance quality and schema compliance.
• The framework supports benchmark ecosystems for multi-hop QA, hallucination detection, and bias-aware evaluation to advance financial AI research.

FinReflectKG is a source-attributed financial knowledge graph framework, dataset, and benchmark ecosystem centered on extracting structured financial knowledge from SEC 10-K filings and linking each extracted fact back to its source evidence. In its initial formulation, it is an open-source, large-scale financial knowledge graph construction framework built from the latest annual SEC 10-K filings of all S&P 100 companies for 2024, combining intelligent document parsing, table-aware chunking, schema-guided iterative extraction, and a reflection-driven feedback loop (Arun et al., 25 Aug 2025). Subsequent work extends the notion of FinReflectKG into a temporally indexed financial KG that links audited triples to source chunks and supports benchmark suites for multi-hop question answering, hallucination detection, and multi-dimensional extraction evaluation (Arun et al., 3 Oct 2025, Kumar et al., 11 Mar 2026, Dimino et al., 7 Oct 2025).

1. Definition, scope, and corpus

Across the FinReflectKG papers, the term denotes more than a static graph. It refers to an upstream KG construction pipeline, a provenance layer linking triples to source chunks, and a family of downstream benchmarks for financial AI. The original system paper presents FinReflectKG as an open-source dataset and agentic construction framework over the latest annual SEC 10-K filings of all S&P 100 companies for 2024 (Arun et al., 25 Aug 2025). The MultiHop benchmark paper describes the broader FinReflectKG dataset as containing 17.5M triplets across 743 S&P 500 companies spanning 2014–2024, with 24 entity types and 29 relationship types, and emphasizes temporal indexing and source grounding to filing chunks (Arun et al., 3 Oct 2025).

This dual description is significant because it shows that FinReflectKG is used both as a concrete released corpus over a fixed filing year and as a larger, temporally indexed graph substrate for benchmark construction. In both settings, provenance is central: later papers repeatedly characterize FinReflectKG as linking extracted or “audited” triples to the original filing chunk, enabling local grounding, retrieval, and error analysis rather than purely schema-level graph population (Arun et al., 3 Oct 2025, Dimino et al., 7 Oct 2025).

Resource	Primary role	Reported scope
FinReflectKG (Arun et al., 25 Aug 2025)	KG construction framework and dataset	S&P 100, 2024 SEC 10-K filings
FinReflectKG - MultiHop (Arun et al., 3 Oct 2025)	Multi-hop QA benchmark	2-hop and 3-hop QA with KG-linked evidence
FinReflectKG - HalluBench / FinBench-QA-Hallucination (Kumar et al., 11 Mar 2026)	Hallucination detection benchmark	755 validated examples from SEC 10-K pages
FinReflectKG - EvalBench (Dimino et al., 7 Oct 2025)	Bias-aware extraction evaluation	Multi-dimensional benchmarking of extraction modes

A common misconception is to treat FinReflectKG solely as a benchmark name. The benchmark papers themselves distinguish between FinReflectKG as the upstream financial KG construction pipeline and derivative resources such as FinBench-QA-Hallucination / HalluBench and FinReflectKG - MultiHop as evaluation layers built on top of that pipeline (Kumar et al., 11 Mar 2026).

2. Construction architecture and extraction formalism

The core FinReflectKG pipeline has four stages: Intelligent Document Parsing, Table-Aware Semantic Chunking, Iterative Prompt / Agent-Driven Triple Extraction, and Robust Evaluation (Arun et al., 25 Aug 2025). Parsing is implemented with Docling, preserving narrative text, tables, and images, although the reported experiments use text mode. Tables are retained as markdown, and sections are tagged with headers such as “Risk Factors” or “Management’s Discussion” (Arun et al., 25 Aug 2025).

Chunking is explicitly semantic and table-aware rather than sliding-window based. Any table is kept as a single atomic chunk; text is split at logical boundaries such as paragraphs and subsection headings; and each chunk is capped at 2048 tokens (Arun et al., 25 Aug 2025). This design is motivated by the structure of 10-K filings, where quantitative disclosures and section-local semantics can be corrupted by naive segmentation.

Extraction is conducted in a closed information extraction regime over a preconfigured schema approved by financial SMEs, with triples represented as

$$(\text{Head Entity},\ \text{Head Type},\ \text{Relationship},\ \text{Tail Entity},\ \text{Tail Type}).$$

The paper explicitly treats these 5-tuples as “triples” in the KG context. Example entity types include ORG, PERSON, COMP, PRODUCT, SEGMENT, FIN_METRIC, RISK_FACTOR, EVENT, REGULATORY_REQUIREMENT, and ESG_TOPIC; example relation types include Has_Stake_In, Operates_In, Produces, Impacts, Involved_In, Impacted_By, Discloses, Complies_With, Supplies, and Partners_With. The LLM used for KG construction is Qwen2.5-72B-Instruct (Arun et al., 25 Aug 2025).

FinReflectKG compares three extraction modes. In single-pass mode, one prompt extracts all valid triples from a chunk in one shot:

$$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{sp}}).$$

In multi-pass mode, extraction is followed by normalization and cleanup:

$$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{mp}}), \qquad T_c^{(2)} = \mathrm{Normalize}(c, T_c^{(1)}, S, \phi_{\mathrm{mp}}).$$

In reflection-agent-based mode, a critic-corrector loop iteratively revises candidate triples:

$$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{re}}),$$

$$F_c^{(t)} = \mathrm{Feedback}(c, T_c^{(t-1)}, S, \phi_{\mathrm{re}}),$$

$$T_c^{(t)} = \mathrm{Correct}(c, T_c^{(t-1)}, F_c^{(t)}, S, \phi_{\mathrm{re}}).$$

The stopping criterion is either $F_c^{(t^*)} = \emptyset$ or $t^* = n_{\max}$, with final chunk output $T_c^{(*)} = T_c^{(t^*)}$, and document-level aggregation

$T_D^{(*)} = \bigcup_{i=1}^N T_{c_i}^{(*)}.$

This formulation is the basis for the “Reflect” component in the name FinReflectKG. The reflection loop is presented as the central innovation of the construction pipeline, intended to improve schema compliance, business relevance, and extraction quality without fine-tuning (Arun et al., 25 Aug 2025).

3. Quality control, judging protocols, and metric design

FinReflectKG is unusual among financial KG systems in that evaluation is treated as a first-class component rather than an afterthought. The original construction paper argues that financial KG evaluation cannot rely on standard ground-truth annotation alone and therefore combines rule-based checks, statistical validation, entropy-based diversity measures, and LLM-as-a-Judge comparison (Arun et al., 25 Aug 2025).

The rule-based layer, CheckRules, verifies four policies for each triple: Subject Reference, Entity Length Constraint, Entity Schema Compliance, and Relationship Schema Compliance. The score for a triple $$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{sp}}).$$0 is

$$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{sp}}).$$1

where $$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{sp}}).$$2 indicates whether rule $$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{sp}}).$$3 is satisfied. Aggregate reporting uses thresholds such as “at least 1 rule” through “all 4 rules” (Arun et al., 25 Aug 2025). Statistical validation adds ECR, TCR, TCR-N, RCR, and RCR-N, as well as triples per chunk. Global semantic diversity is measured with Shannon entropy, schema-normalized entropy, and Rényi entropy with $$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{sp}}).$$4 (Arun et al., 25 Aug 2025).

The later FinReflectKG - EvalBench paper systematizes this into a unified benchmark architecture. It formalizes chunk-level extraction as $$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{sp}}).$$5, where each candidate triple is evaluated against its source chunk using a deterministic LLM judge instantiated with Qwen3-32B at temperature = 0.0 (Dimino et al., 7 Oct 2025). EvalBench adopts a deterministic commit-then-justify protocol: the judge first emits a structured verdict and then a brief justification in JSON, with fields "verdict": 0|1, "reasoning": "...", and "warning": "...", where the reasoning is capped at 15 words (Dimino et al., 7 Oct 2025).

EvalBench’s most distinctive feature is explicit bias control. The judge is instructed to default to 0 under ambiguity, to avoid outside knowledge, to ignore sentence order and text position, and not to let verbosity or surface form affect the decision. Few-shot examples are provided for each criterion (Dimino et al., 7 Oct 2025). The evaluation dimensions are Faithfulness, Precision, Relevance, and Comprehensiveness. Faithfulness, precision, and relevance are binary; comprehensiveness is a chunk-level ordinal variable with good, partial, and bad, encoded as 3, 2, and 1 respectively (Dimino et al., 7 Oct 2025).

The significance of this metric design is methodological. FinReflectKG treats financial KG quality as irreducibly multi-dimensional: a system can be highly grounded yet incomplete, or comprehensive yet somewhat less faithful. This trade-off is repeatedly emphasized across the original construction paper and EvalBench (Arun et al., 25 Aug 2025, Dimino et al., 7 Oct 2025).

4. Benchmark ecosystem built on FinReflectKG

The benchmark family built on FinReflectKG currently has three major components: a hallucination benchmark, a multi-hop QA benchmark, and a bias-aware extraction benchmark. Together they operationalize three different uses of source-grounded financial KGs: reliability testing, retrieval-focused reasoning, and extraction assessment.

FinReflectKG - HalluBench, whose body text standardizes the benchmark name as FinBench-QA-Hallucination, evaluates hallucination detection in KG-augmented financial QA over SEC 10-K filings (Kumar et al., 11 Mar 2026). Its construction pipeline extracts KG triplets from filing pages using FinReflectKG, filters to pages with at least 10 extracted triplets, samples 5–7 triplets per page, uses Qwen-3-235B to generate (Question, Answer, Reasoning) tuples, applies an automated LLM-as-judge filter with a “When in doubt, DROP” rule, and manually validates the remainder. The final dataset contains 755 validated examples from 300 pages and 57 unique tickers, with 513 grounded and 242 hallucinated examples (Kumar et al., 11 Mar 2026).

Its definition of groundedness is deliberately conservative:

$$T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{sp}}).$$6

This protocol is central to HalluBench’s role within the FinReflectKG ecosystem: it tests not only whether an answer is factually right, but whether it is evidentially licensed by both text and graph evidence (Kumar et al., 11 Mar 2026).

FinReflectKG - MultiHop targets analyst-style multi-hop financial QA (Arun et al., 3 Oct 2025). It mines frequent 2-hop and 3-hop subgraph patterns across sectors using GICS taxonomy, converts them into natural-language questions, and evaluates retrieval under three controlled scenarios: S1 exact KG-linked evidence, S2 ±5 page window evidence, and S3 page-window evidence with randomizations and distractors. The released subset contains 555 QA pairs, while the paper reports evaluation on a top 150 QA subset spanning the Financials and Information Technology sectors (Arun et al., 3 Oct 2025).

FinReflectKG - EvalBench serves a different purpose: it is not a QA benchmark but a benchmark and evaluation framework for KG extraction from SEC 10-K filings, explicitly designed to make LLM-based judging reproducible, conservative, and bias-aware (Dimino et al., 7 Oct 2025). It inherits the single-pass, multi-pass, and reflection-agent-based extraction settings from the main FinReflectKG construction framework and provides a common evaluation interface over source-linked triples.

5. Empirical results and characteristic trade-offs

The main empirical finding of the original FinReflectKG paper is that the reflection-agent-based mode gives the strongest overall quality–coverage balance (Arun et al., 25 Aug 2025). Under the strictest CheckRules threshold, requiring satisfaction of all four rules, the reported compliance scores are 42.3% for single-pass, 47.3% for multi-pass, and 64.8% for reflection. Reflection also leads on extraction richness, with 15.8 triples per chunk versus 13.3 and 12.4, ECR = 0.53 versus 0.30 and 0.31, and RCR = 0.38 versus 0.21 and 0.22 (Arun et al., 25 Aug 2025).

The same paper reports a more nuanced pattern under LLM-as-a-Judge comparison. Reflection is best on Precision (39.1), Comprehensiveness (48.1), and Relevance (37.3), while single-pass is highest on Faithfulness (40.1) (Arun et al., 25 Aug 2025). EvalBench reproduces the same qualitative trade-off with a more formal judging setup: Reflection reaches 72.01 on comprehensiveness, 59.49 on precision, and 92.52 on relevance, whereas Single Pass achieves the highest faithfulness at 87.25 (Dimino et al., 7 Oct 2025). The consistent interpretation across both papers is that iterative reflection increases coverage and topical adequacy but can introduce some drift relative to the most conservative extraction mode.

HalluBench exposes a different trade-off: KG augmentation is informative under clean evidence but fragile under KG noise (Kumar et al., 11 Mar 2026). In text-only and text-plus-triplets conditions, Qwen LLM judge attains F1 0.863 without triplets and 0.818 with triplets; Qwen embeddings attain 0.824 in both conditions; and most methods show substantial MCC degradation when noisy triplets are introduced. The paper states that MCC drops by 44–84% across methods, while embedding methods show only about 9% degradation. Statistical tests report Cochran’s Q significance in both conditions, with Q = 132.63, p < 0.001 without triplets and Q = 73.85, p < 0.001 with triplets (Kumar et al., 11 Mar 2026).

MultiHop, by contrast, demonstrates the value of precise KG-guided retrieval. The headline result is that KG-linked evidence improves correctness by about 24% on average over the page-window setting while reducing input tokens by about 84.5% (Arun et al., 3 Oct 2025). Reported examples include Qwen3-32B with 8.23 versus 6.59 correctness and 2069 versus 13602 input tokens, and GPT-OSS-120B with 8.09 versus 7.12 correctness and 1967 versus 12414 input tokens. The appendix reports correctness lifts ranging from 13.6% to 39.2% depending on model size, with smaller models benefiting more from KG-guided retrieval (Arun et al., 3 Oct 2025).

Taken together, these results give FinReflectKG a characteristic empirical profile: reflection improves extraction coverage, KG-linked retrieval improves financial QA efficiency and correctness, but structured evidence can also mislead downstream systems when graph inputs are noisy or weakly verified.

6. Applications, governance implications, and relation to adjacent financial KG work

The FinReflectKG papers frame the system as infrastructure for high-stakes financial AI. Reported downstream or intended use cases include compliance monitoring, risk management, investment research, credit analysis, multi-hop QA, entity search over filings, and more auditable financial NLP systems (Arun et al., 25 Aug 2025, Dimino et al., 7 Oct 2025). HalluBench explicitly motivates hallucination detection by the risk of “compliance checks, investment analysis, regulatory reporting” being affected by small errors in numbers, units, or time periods (Kumar et al., 11 Mar 2026).

The governance contribution lies in source linkage, conservative judging, and benchmarkable failure analysis. FinReflectKG is designed so that a claim can be traced back to a filing chunk, tested for groundedness, and evaluated under controlled retrieval or judging conditions. This makes it not merely a repository of extracted facts but a framework for evidential accountability in financial AI (Dimino et al., 7 Oct 2025, Kumar et al., 11 Mar 2026).

Within the broader literature, FinReflectKG belongs to a larger movement from unstructured financial text toward graph-based, provenance-aware reasoning. FinDKG constructs a dynamic financial news KG from approximately 400,000 Wall Street Journal articles spanning 1999 to 2023 and pairs it with KGTransformer for link prediction and thematic investing, emphasizing temporal evolution rather than filing-grounded audit trails (Li et al., 2024). FinKG-News builds a news-centric financial KG connecting companies and historical events, reaching 80,418 entities and 261,168 triples, and uses that graph as grounded evidence for credit risk report generation, where the proposed method improves report quality by 19%–34% while reducing hallucinations (Jimenez-Villen et al., 1 Jul 2026).

This suggests a useful conceptual placement for FinReflectKG. It is the filing-centric, provenance-heavy branch of financial KG research: less focused on market-reflection dynamics than FinDKG, less focused on event-centric news grounding than FinKG-News, and more focused on auditable extraction, benchmark design, and reliable evidence retrieval from corporate disclosures (Li et al., 2024, Jimenez-Villen et al., 1 Jul 2026). In that sense, FinReflectKG functions both as a dataset and as a methodological template for building financial AI systems that are traceable, benchmarkable, and explicitly grounded in source documents.