- The paper introduces FinKG-News, a pipeline that links 354 curated global events to 12,407 companies through news evidence, producing 80,418 entities and 261,168 triples with event detection reaching a 0.72 micro-F1 score.
- The paper decomposes credit analysis into financial profile, ownership record, and operating environment drivers, then combines peer comparison, graph-grounded evidence, reflection prompting, and report ensembling to improve expert-rated quality by 19–34% across five companies.
- The paper shows that evidence grounding can reduce hallucination and make reasoning more verifiable, while highlighting important limitations including weak automatic evaluation, co-occurrence-based impact relations, event-detection errors, scalability costs, and limited five-company validation.
FinKG-News addresses two coupled problems in financial natural language processing: the absence of real-world event dynamics in existing financial knowledge graphs (FKGs), and the severe hallucination rates that LLMs exhibit when generating domain reports for companies with sparse public reference data. The authors, from Universidad Politécnica de Madrid and Institute of Science Tokyo, propose a news-centric knowledge graph construction pipeline and an in-context learning architecture for credit risk report generation, evaluated against expert and automatic assessment (2607.01023).
Motivation and positioning
Existing FKGs model static corporate structure — equity ownership in FinKG, supply chains in Refinitiv's knowledge base — but do not encode the event–market relationships that drive market dynamics. Text-extracted FKGs such as FinDKG and FinReflectKG leave event and company coverage implicit, making entity coverage dependent on the extraction model. FinKG-News instead anchors the graph on a fixed, curated inventory of historical events and links them to companies through news evidence, explicitly targeting small- and mid-cap firms where LLM hallucination is most acute due to limited training coverage.
Knowledge graph construction
The construction pipeline has three stages. First, an event inventory of 354 major global events (2000–2025) is drawn from the Wikipedia Timeline of the 21st Century, each annotated with a description, date or interval, geographical scope, and a category from a three-level taxonomy (political/non-political at the top level, refined to leaf categories such as inter-state wars, health crises, or disruptive technologies). The annotations were produced by GPT-4.1 and iteratively refined using quality ratings from a second, independent model (Gemini-2.5-flash).
Second, event detection matches incoming news articles against this fixed event list. The event inventory is chunked (N=20 per prompt) and injected into the system prompt of Llama3:70B, which outputs matched event identifiers with confidence scores; a temporal filter restricts candidates to events on or before the article date. Post-processing discards detections below a 0.9 confidence threshold, and heuristic hallucination checks remove fabricated identifiers, malformed outputs, and references outside the predefined list.
Third, graph population extends FinKG with News, Event, and Event Category entities. Event–IMPACTS–Company relations are inferred co-occurrence-style: an event is linked to companies mentioned in the same article via News–MENTIONS–Event and News–MENTIONS–Company triples. The resulting graph contains 80,418 entities and 261,168 triples overall, with the news-derived portion contributing 2,428 news nodes, 354 events, 6,874 MENTIONS triples, and 2,391 IMPACTS triples across 12,407 companies spanning large-, mid-, and small-capitalization segments.
On a manually annotated test set of 100 articles, framed as multi-label classification over 354 event classes, the detection component achieves a micro F1-score of 0.72 (precision 0.694, recall 0.756) and exact match accuracy of 0.64. The gap between micro (0.52 F1 macro) and micro-averaged scores indicates uneven performance across event classes, which the authors attribute to news frequency imbalance. This is a moderate detection ceiling: roughly a third of articles are not perfectly matched, and downstream report quality inherits whatever noise this component introduces.
Report generation architecture
The generation pipeline follows a simplified version of Fitch Ratings' corporate rating criteria, decomposed into three key rating drivers: F1 (Financial Profile) — quantitative indicators such as EBITDA, total debt, and free cash flow over a four-year window, sourced from public data; F2 (Ownership Record) — SEC insider and issuer transaction tables plus parent–subsidiary structure queried from FinKG-News via IS_PARTIAL_OWNER relations; and F3 (Operating Environment) — a subgraph centered on the target company containing all linked events, the company's Standard Industrial Classification, sector peers with events, and event categories.
Three techniques shape the LLM's behavior. Peer comparison grounds F1 and F3 in a same-industry peer selected by asset-size similarity, providing relative rather than absolute context. In-context learning is applied per driver, with guiding questions designed with a financial expert to force specific, non-generic analysis. Reflection prompting prompts the model to self-revise for hallucinations, missing risks, and ambiguity, drawing on prior evidence of LLM self-correction capability. Finally, the three per-driver reports are ensembled into a single output, following the chain-of-thought principle that task decomposition improves accuracy and interpretability.
Evaluation results
Five SMEs (Alaska Air, CoreCivic, Disney, Merck, Occidental Petroleum) were evaluated by financial experts and an automatic scoring framework, each on content quality and hallucination-detection indicators normalized to [0,1]. The central quantitative findings:
| Company |
Human baseline |
Human (ours) |
Human improvement |
Auto improvement |
| Alaska Air |
0.653 |
0.838 |
0.185 |
0.015 |
| CoreCivic |
0.500 |
0.840 |
0.340 |
0.105 |
| Disney |
0.659 |
0.852 |
0.193 |
0.165 |
| Merck & Co |
0.668 |
0.863 |
0.195 |
0.000 |
| Occidental Petroleum |
0.536 |
0.861 |
0.325 |
0.121 |
Expert-rated improvements range from 18% to 34%, while automatic scoring yields smaller and inconsistent gains — including zero improvement for Merck. For the firm with the largest expert uplift (CoreCivic, +34%), indicator-level results show Usefulness +0.65, Attention +0.70, and Correctness and Fabrication each +0.65 under human evaluation, indicating the gains stem from traceable, evidence-grounded analysis rather than stylistic polish. Two indicators decline: Balance (−0.05) and Ecological Validity (−0.20), a trade-off the authors attribute to the system making explicit conditional predictions that replace the baseline's conservative or tautological statements.
A notable and somewhat contradictory result concerns the automatic evaluator: it is systematically less sensitive to hallucinations than expert reviewers, and on Fabrication detection it actually scores the generated reports lower than the baseline (−0.35) even as experts score them higher (+0.65). The authors conclude that automated hallucination detection and quality assessment remain unreliable and that expert judgment is indispensable — a claim that cuts against the common practice of relying on LLM-as-judge evaluation in this domain.
Qualitative expert feedback identifies three mechanisms behind the improvements: peer benchmarks that contextualize indicator changes and reduce emotionally charged language; integration of external signals (news events, insider trading, governance indicators) absent from baseline reports; and explicit reasoning chains that make conclusions verifiable. The authors give a concrete example where the baseline conflates declining profitability with higher refinancing risk without the necessary auxiliary conditions, whereas the grounded report states the full conditional logic.
Relation to prior work
The paper extends FinKG, which prioritizes high-quality structured SEC data, with event-centric information from news. Compared to LLM-generated FKGs such as FinDKG (news-derived, schema-guided) and FinReflectKG (10-K-derived), the fixed event inventory provides concrete, sometimes less well-known entities — particularly benefiting smaller-cap firms — at the cost of coverage being bounded by the curated event list. The work also relates to knowledge-augmented financial report generation (Chen et al.) and to news-driven credit risk monitoring (NRAS), differing in the explicit graph-mediated evidence structure and the three-driver decomposition aligned with rating agency methodology.
Limitations and open questions
The authors identify two principal limitations. The event-chunking workaround for context-window constraints multiplies LLM calls per article, raising computation cost and questioning scalability to larger event inventories. More substantively, the IMPACTS relation is inferred by co-occurrence without typed or causal semantics; when the nature of an event's impact on a company is not evident from the article, the LLM defaults to generic or ambiguous explanations in the generated reports. The authors propose enriching Event–Company relations with causal or contextual evidence extracted from the same articles, or adding a causality-check module during triple inference, as open directions. Two further caveats bear on interpretation: the evaluation covers only five firms with a small expert panel, and the event detection ceiling (micro F1 0.72) means a nontrivial fraction of relevant events may never enter the graph.
Conclusion
FinKG-News demonstrates that anchoring a financial knowledge graph on a curated, identifier-resolved event inventory and grounding report generation in graph-retrieved evidence yields substantial expert-rated quality gains (19–34%) with reduced hallucination for credit risk reporting, including for small- and mid-cap firms where LLMs are least reliable. Its evaluation contributes a cautionary finding: automatic judges underperform human experts at hallucination detection in this setting, and can even invert the direction of the Fabrication signal. The open questions the paper leaves — causal typing of event–company impacts, scalability of chunked detection, and validation beyond five firms — are concrete and well-posed.
