---
title: 'FinReflectKG: Financial Knowledge Graph Framework'
url: https://www.emergentmind.com/topics/finreflectkg
type: topic
---

# FinReflectKG: Financial Knowledge Graph Framework

FinReflectKG is a source-attributed financial knowledge graph framework, dataset, and benchmark ecosystem centered on extracting structured financial knowledge from SEC 10-K filings and linking each extracted fact back to its source evidence. In its initial formulation, it is an open-source, large-scale financial knowledge graph construction framework built from the latest annual SEC 10-K filings of all S&P 100 companies for 2024, combining intelligent document parsing, table-aware chunking, schema-guided iterative extraction, and a reflection-driven feedback loop [2508.17906]. Subsequent work extends the notion of FinReflectKG into a temporally indexed financial KG that links audited triples to source chunks and supports benchmark suites for multi-hop question answering, hallucination detection, and multi-dimensional extraction evaluation [2510.02906][2603.20252][2510.05710].

## 1. Definition, scope, and corpus

Across the FinReflectKG papers, the term denotes more than a static graph. It refers to an upstream KG construction pipeline, a provenance layer linking triples to source chunks, and a family of downstream benchmarks for financial AI. The original system paper presents FinReflectKG as an open-source dataset and agentic construction framework over the latest annual SEC 10-K filings of all S&P 100 companies for 2024 [2508.17906]. The MultiHop benchmark paper describes the broader FinReflectKG dataset as containing **17.5M triplets across 743 S&P 500 companies spanning 2014–2024**, with **24 entity types** and **29 relationship types**, and emphasizes temporal indexing and source grounding to filing chunks [2510.02906].

This dual description is significant because it shows that FinReflectKG is used both as a concrete released corpus over a fixed filing year and as a larger, temporally indexed graph substrate for benchmark construction. In both settings, provenance is central: later papers repeatedly characterize FinReflectKG as linking extracted or “audited” triples to the original filing chunk, enabling local grounding, retrieval, and error analysis rather than purely schema-level graph population [2510.02906][2510.05710].

| Resource | Primary role | Reported scope |
|---|---|---|
| FinReflectKG [2508.17906] | KG construction framework and dataset | S&P 100, 2024 SEC 10-K filings |
| FinReflectKG - MultiHop [2510.02906] | Multi-hop QA benchmark | 2-hop and 3-hop QA with KG-linked evidence |
| FinReflectKG - HalluBench / FinBench-QA-Hallucination [2603.20252] | Hallucination detection benchmark | 755 validated examples from SEC 10-K pages |
| FinReflectKG - EvalBench [2510.05710] | Bias-aware extraction evaluation | Multi-dimensional benchmarking of extraction modes |

A common misconception is to treat FinReflectKG solely as a benchmark name. The benchmark papers themselves distinguish between **FinReflectKG** as the upstream financial KG construction pipeline and derivative resources such as **FinBench-QA-Hallucination / HalluBench** and **FinReflectKG - MultiHop** as evaluation layers built on top of that pipeline [2603.20252].

## 2. Construction architecture and extraction formalism

The core FinReflectKG pipeline has four stages: **Intelligent Document Parsing**, **Table-Aware Semantic Chunking**, **Iterative Prompt / Agent-Driven Triple Extraction**, and **Robust Evaluation** [2508.17906]. Parsing is implemented with **Docling**, preserving narrative text, tables, and images, although the reported experiments use **text mode**. Tables are retained as markdown, and sections are tagged with headers such as “Risk Factors” or “Management’s Discussion” [2508.17906].

Chunking is explicitly semantic and table-aware rather than sliding-window based. Any table is kept as a **single atomic chunk**; text is split at **logical boundaries** such as paragraphs and subsection headings; and each chunk is capped at **2048 tokens** [2508.17906]. This design is motivated by the structure of 10-K filings, where quantitative disclosures and section-local semantics can be corrupted by naive segmentation.

Extraction is conducted in a **closed information extraction** regime over a preconfigured schema approved by financial SMEs, with triples represented as

\[
(\text{Head Entity},\ \text{Head Type},\ \text{Relationship},\ \text{Tail Entity},\ \text{Tail Type}).
\]

The paper explicitly treats these 5-tuples as “triples” in the KG context. Example entity types include `ORG`, `PERSON`, `COMP`, `PRODUCT`, `SEGMENT`, `FIN_METRIC`, `RISK_FACTOR`, `EVENT`, `REGULATORY_REQUIREMENT`, and `ESG_TOPIC`; example relation types include `Has_Stake_In`, `Operates_In`, `Produces`, `Impacts`, `Involved_In`, `Impacted_By`, `Discloses`, `Complies_With`, `Supplies`, and `Partners_With`. The LLM used for KG construction is **Qwen2.5-72B-Instruct** [2508.17906].

FinReflectKG compares three extraction modes. In **single-pass mode**, one prompt extracts all valid triples from a chunk in one shot:

\[
T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{sp}}).
\]

In **multi-pass mode**, extraction is followed by normalization and cleanup:

\[
T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{mp}}), \qquad
T_c^{(2)} = \mathrm{Normalize}(c, T_c^{(1)}, S, \phi_{\mathrm{mp}}).
\]

In **reflection-agent-based mode**, a critic-corrector loop iteratively revises candidate triples:

\[
T_c^{(1)} = \mathrm{Extract}(c, S, \phi_{\mathrm{re}}),
\]
\[
F_c^{(t)} = \mathrm{Feedback}(c, T_c^{(t-1)}, S, \phi_{\mathrm{re}}),
\]
\[
T_c^{(t)} = \mathrm{Correct}(c, T_c^{(t-1)}, F_c^{(t)}, S, \phi_{\mathrm{re}}).
\]

The stopping criterion is either \(F_c^{(t^*)} = \emptyset\) or \(t^* = n_{\max}\), with final chunk output \(T_c^{(*)} = T_c^{(t^*)}\), and document-level aggregation

\[
T_D^{(*)} = \bigcup_{i=1}^N T_{c_i}^{(*)}.
\]

This formulation is the basis for the “Reflect” component in the name FinReflectKG. The reflection loop is presented as the central innovation of the construction pipeline, intended to improve schema compliance, business relevance, and extraction quality without fine-tuning [2508.17906].

## 3. Quality control, judging protocols, and metric design

FinReflectKG is unusual among financial KG systems in that evaluation is treated as a first-class component rather than an afterthought. The original construction paper argues that financial KG evaluation cannot rely on standard ground-truth annotation alone and therefore combines rule-based checks, statistical validation, entropy-based diversity measures, and LLM-as-a-Judge comparison [2508.17906].

The rule-based layer, **CheckRules**, verifies four policies for each triple: **Subject Reference**, **Entity Length Constraint**, **Entity Schema Compliance**, and **Relationship Schema Compliance**. The score for a triple \(t\) is

\[
CR(t)=\frac{1}{R}\sum_{i=1}^{R}\phi_i(t),
\]

where \(\phi_i(t)\in\{0,1\}\) indicates whether rule \(i\) is satisfied. Aggregate reporting uses thresholds such as “at least 1 rule” through “all 4 rules” [2508.17906]. Statistical validation adds **ECR**, **TCR**, **TCR-N**, **RCR**, and **RCR-N**, as well as triples per chunk. Global semantic diversity is measured with **Shannon entropy**, schema-normalized entropy, and **Rényi entropy** with \(\alpha=2\) [2508.17906].

The later **FinReflectKG - EvalBench** paper systematizes this into a unified benchmark architecture. It formalizes chunk-level extraction as \(T_x = E(x) \subseteq \mathcal{T}\), where each candidate triple is evaluated against its source chunk using a deterministic LLM judge instantiated with **Qwen3-32B** at **temperature = 0.0** [2510.05710]. EvalBench adopts a **deterministic commit-then-justify** protocol: the judge first emits a structured verdict and then a brief justification in JSON, with fields `"verdict": 0|1`, `"reasoning": "..."`, and `"warning": "..."`, where the reasoning is capped at **15 words** [2510.05710].

EvalBench’s most distinctive feature is explicit bias control. The judge is instructed to default to **0** under ambiguity, to avoid outside knowledge, to ignore sentence order and text position, and not to let verbosity or surface form affect the decision. Few-shot examples are provided for each criterion [2510.05710]. The evaluation dimensions are **Faithfulness**, **Precision**, **Relevance**, and **Comprehensiveness**. Faithfulness, precision, and relevance are binary; comprehensiveness is a chunk-level ordinal variable with **good**, **partial**, and **bad**, encoded as **3**, **2**, and **1** respectively [2510.05710].

The significance of this metric design is methodological. FinReflectKG treats financial KG quality as irreducibly multi-dimensional: a system can be highly grounded yet incomplete, or comprehensive yet somewhat less faithful. This trade-off is repeatedly emphasized across the original construction paper and EvalBench [2508.17906][2510.05710].

## 4. Benchmark ecosystem built on FinReflectKG

The benchmark family built on FinReflectKG currently has three major components: a hallucination benchmark, a multi-hop QA benchmark, and a bias-aware extraction benchmark. Together they operationalize three different uses of source-grounded financial KGs: reliability testing, retrieval-focused reasoning, and extraction assessment.

**FinReflectKG - HalluBench**, whose body text standardizes the benchmark name as **FinBench-QA-Hallucination**, evaluates hallucination detection in KG-augmented financial QA over SEC 10-K filings [2603.20252]. Its construction pipeline extracts KG triplets from filing pages using FinReflectKG, filters to pages with at least **10 extracted triplets**, samples **5–7 triplets per page**, uses **Qwen-3-235B** to generate **(Question, Answer, Reasoning)** tuples, applies an automated **LLM-as-judge** filter with a “**When in doubt, DROP**” rule, and manually validates the remainder. The final dataset contains **755** validated examples from **300 pages** and **57 unique tickers**, with **513 grounded** and **242 hallucinated** examples [2603.20252].

Its definition of groundedness is deliberately conservative:

\[
\text{Grounded} \iff \text{correct w.r.t. source chunk} \land \text{supported by supporting triplet} \land \text{supported by supporting lines}.
\]

This protocol is central to HalluBench’s role within the FinReflectKG ecosystem: it tests not only whether an answer is factually right, but whether it is evidentially licensed by both text and graph evidence [2603.20252].

**FinReflectKG - MultiHop** targets analyst-style multi-hop financial QA [2510.02906]. It mines frequent **2-hop** and **3-hop** subgraph patterns across sectors using **GICS taxonomy**, converts them into natural-language questions, and evaluates retrieval under three controlled scenarios: **S1** exact KG-linked evidence, **S2** **±5 page window** evidence, and **S3** page-window evidence with randomizations and distractors. The released subset contains **555 QA pairs**, while the paper reports evaluation on a **top 150 QA subset** spanning the Financials and Information Technology sectors [2510.02906].

**FinReflectKG - EvalBench** serves a different purpose: it is not a QA benchmark but a benchmark and evaluation framework for KG extraction from SEC 10-K filings, explicitly designed to make LLM-based judging reproducible, conservative, and bias-aware [2510.05710]. It inherits the single-pass, multi-pass, and reflection-agent-based extraction settings from the main FinReflectKG construction framework and provides a common evaluation interface over source-linked triples.

## 5. Empirical results and characteristic trade-offs

The main empirical finding of the original FinReflectKG paper is that the **reflection-agent-based mode** gives the strongest overall quality–coverage balance [2508.17906]. Under the strictest CheckRules threshold, requiring satisfaction of all four rules, the reported compliance scores are **42.3%** for single-pass, **47.3%** for multi-pass, and **64.8%** for reflection. Reflection also leads on extraction richness, with **15.8** triples per chunk versus **13.3** and **12.4**, **ECR = 0.53** versus **0.30** and **0.31**, and **RCR = 0.38** versus **0.21** and **0.22** [2508.17906].

The same paper reports a more nuanced pattern under LLM-as-a-Judge comparison. Reflection is best on **Precision** (**39.1**), **Comprehensiveness** (**48.1**), and **Relevance** (**37.3**), while single-pass is highest on **Faithfulness** (**40.1**) [2508.17906]. EvalBench reproduces the same qualitative trade-off with a more formal judging setup: **Reflection** reaches **72.01** on comprehensiveness, **59.49** on precision, and **92.52** on relevance, whereas **Single Pass** achieves the highest faithfulness at **87.25** [2510.05710]. The consistent interpretation across both papers is that iterative reflection increases coverage and topical adequacy but can introduce some drift relative to the most conservative extraction mode.

HalluBench exposes a different trade-off: KG augmentation is informative under clean evidence but fragile under KG noise [2603.20252]. In text-only and text-plus-triplets conditions, **Qwen LLM judge** attains **F1 0.863** without triplets and **0.818** with triplets; **Qwen embeddings** attain **0.824** in both conditions; and most methods show substantial **MCC** degradation when noisy triplets are introduced. The paper states that MCC drops by **44–84% across methods**, while embedding methods show only about **9% degradation**. Statistical tests report **Cochran’s Q** significance in both conditions, with **Q = 132.63, p < 0.001** without triplets and **Q = 73.85, p < 0.001** with triplets [2603.20252].

MultiHop, by contrast, demonstrates the value of precise KG-guided retrieval. The headline result is that KG-linked evidence improves correctness by about **24% on average** over the page-window setting while reducing input tokens by about **84.5%** [2510.02906]. Reported examples include **Qwen3-32B** with **8.23** versus **6.59** correctness and **2069** versus **13602** input tokens, and **GPT-OSS-120B** with **8.09** versus **7.12** correctness and **1967** versus **12414** input tokens. The appendix reports correctness lifts ranging from **13.6%** to **39.2%** depending on model size, with smaller models benefiting more from KG-guided retrieval [2510.02906].

Taken together, these results give FinReflectKG a characteristic empirical profile: reflection improves extraction coverage, KG-linked retrieval improves financial QA efficiency and correctness, but structured evidence can also mislead downstream systems when graph inputs are noisy or weakly verified.

## 6. Applications, governance implications, and relation to adjacent financial KG work

The FinReflectKG papers frame the system as infrastructure for high-stakes financial AI. Reported downstream or intended use cases include **compliance monitoring**, **risk management**, **investment research**, **credit analysis**, **multi-hop QA**, **entity search over filings**, and more auditable financial NLP systems [2508.17906][2510.05710]. HalluBench explicitly motivates hallucination detection by the risk of “compliance checks, investment analysis, regulatory reporting” being affected by small errors in numbers, units, or time periods [2603.20252].

The governance contribution lies in source linkage, conservative judging, and benchmarkable failure analysis. FinReflectKG is designed so that a claim can be traced back to a filing chunk, tested for groundedness, and evaluated under controlled retrieval or judging conditions. This makes it not merely a repository of extracted facts but a framework for evidential accountability in financial AI [2510.05710][2603.20252].

Within the broader literature, FinReflectKG belongs to a larger movement from unstructured financial text toward graph-based, provenance-aware reasoning. **FinDKG** constructs a dynamic financial news KG from approximately **400,000 Wall Street Journal articles spanning 1999 to 2023** and pairs it with **KGTransformer** for link prediction and thematic investing, emphasizing temporal evolution rather than filing-grounded audit trails [2407.10909]. **FinKG-News** builds a news-centric financial KG connecting companies and historical events, reaching **80,418 entities** and **261,168 triples**, and uses that graph as grounded evidence for credit risk report generation, where the proposed method improves report quality by **19%–34%** while reducing hallucinations [2607.01023].

This suggests a useful conceptual placement for FinReflectKG. It is the filing-centric, provenance-heavy branch of financial KG research: less focused on market-reflection dynamics than FinDKG, less focused on event-centric news grounding than FinKG-News, and more focused on auditable extraction, benchmark design, and reliable evidence retrieval from corporate disclosures [2407.10909][2607.01023]. In that sense, FinReflectKG functions both as a dataset and as a methodological template for building financial AI systems that are traceable, benchmarkable, and explicitly grounded in source documents.

Source: https://www.emergentmind.com/topics/finreflectkg