FinRED: Dual Finance Benchmark Resources

• FinRED is a dual-use label for finance evaluation, covering both a relation extraction dataset and a red-teaming framework.
• The 2023 FinRED employs distant supervision on financial news and earnings call transcripts with expert cleaning to benchmark relation extraction.
• The 2026 FinRED uses a schema-driven, expert-guided approach to generate adversarial prompts for evaluating LLM safety in finance.

FinRED is a name used for two distinct finance-domain benchmark resources in recent arXiv literature. The first, introduced in 2023, is a relation extraction dataset curated from financial news and earnings call transcripts to address the absence of an English benchmark focused specifically on financial relations. The second, introduced in 2026 in the paper titled "FFinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming," is an expert-guided framework for generating and evaluating finance-specific adversarial prompts for LLMs. Despite the shared label, the two resources target different technical problems: one is a sentence-level triplet extraction benchmark, and the other is a financial AI safety evaluation stack (Sharma et al., 2023, Kim et al., 18 Jun 2026).

1. Scope and dual usage

In the literature, FinRED does not denote a single unified artifact. It denotes two separate resources that emerged from different problem settings in financial NLP and AI safety.

Aspect	2023 FinRED	2026 FinRED
Primary task	Relation extraction in the financial domain	Financial LLM red-teaming and safety evaluation
Core artifact	Distantly supervised dataset with a manually cleaned test set	Expert-guided benchmark generation and evaluation framework
Basic unit	Sentences aligned to relation triplets	Behavioral prompts ("seeds") and a finance-specific judge rubric
Main source material	Financial news and earnings call transcripts	Financial and regulatory documents selected and reviewed by experts
Access model	Released dataset	Gated for qualified researchers

The two usages share a common motivation: both argue that general-purpose benchmarks are inadequate proxies for finance. In the 2023 work, the central issue is relation-set mismatch between source and target domains. In the 2026 work, the analogous issue is that generic safety benchmarks miss finance-specific harms such as regulatory evasion, fraud facilitation, consumer-rights violations, and ICT compliance evasion. This suggests a broader pattern in which FinRED functions less as a single benchmark family than as a recurring label for finance-specific evaluation infrastructure.

2. FinRED as a financial relation extraction dataset

The 2023 FinRED was introduced to fill what its authors describe as a basic but important gap in information extraction research: while relation extraction systems had been benchmarked largely on general-domain datasets such as FB-NYT and WebNLG, there was no comparable open English benchmark focused specifically on financial relations. The dataset is positioned as a response to domain mismatch, since models trained on generic relation inventories may not capture the entities, relation semantics, and textual realizations that matter in finance (Sharma et al., 2023).

FinRED is built from two text sources. The first is a corpus of Webhose financial news containing 47,851 English financial news articles crawled between July 2015 and October 2015. The second is a corpus of earnings call transcripts consisting of 4,713 transcripts collected from seekingalpha.com, dated from June 2019 to September 2019. For the earnings call transcript corpus, both presentation and questionnaire portions are used; presentation segments and answers are treated as “monologues,” and monologues shorter than 200 characters are removed. The paper reports about 200K monologues, 1.8M sentences, and an average of 7.19 sentences per monologue. It also reports “152K tokens in the corpus,” though the paper itself leaves room to read this as a reporting inconsistency relative to the other corpus statistics.

The relation inventory is derived from Wikidata. The authors extract a subset containing 29 manually filtered financial relations and use it as the knowledge base for distant supervision. A defining property of the dataset is that all 29 relation labels are intended to be financial. In the comparative table reported in the paper, FB-NYT has 24 relations of which 4 are financial, WebNLG has 216 relations of which 12 are financial, and FinRED has 29 relations of which all 29 are financial.

The final dataset contains 7,775 sentences split into 5,699 training instances, 1,008 development instances, and 1,068 test instances. Only 920 sentences in the final dataset come from earnings call transcripts, which the authors attribute to the conversational nature of earnings calls, where a sentence often does not explicitly express a Wikidata triplet. Representative relations shown in the paper include “chief executive officer,” “product or material produced,” and “headquarters location,” with example triplets such as $$(\text{Anthony Jenkins}, \text{chief executive officer}, \text{Barclays})$$, $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$, and $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$.

3. Construction, distant supervision, and annotation protocol

The construction methodology follows distant supervision in the style of mapping knowledge-base triplets to text. The paper states that a subset of Wikidata containing 29 financial relations is used as a source of relational triplets and that these triplets are aligned to the financial corpora using the idea of distance supervision. In effect, if a sentence mentions the head and tail entities from a Wikidata fact, that sentence is treated as expressing the corresponding relation (Sharma et al., 2023).

A major issue in this process is weak-supervision noise. The authors report that the initial pipeline yields about 21,000 sentences, but many are noisy, and after filtering they retain 7,775. The most concrete example of noise is the spurious tuple $$(\text{More Than}, \text{industry}, \text{finance})$$, produced from a sentence containing the ordinary phrase “more than.” Because “More Than” is also a Wikidata entity corresponding to an insurance company, a false entity match occurred. The authors state that they studied a small representation of the data in detail and removed datapoints with incorrect entities such as “More Than” and “industry.” The paper does not report more elaborate denoising methods such as multi-instance learning or probabilistic label correction; the cleanup is described primarily as heuristic and manual filtering of noisy distantly supervised examples.

The test set was manually annotated to support reliable evaluation. Two non-native but fluent English speakers annotated the data, and each annotator went through the dataset twice. The reported rule is that all triplets marked incorrect by both annotators were removed. Inter-annotator agreement, measured with Cohen’s $\kappa$, is 82.1%, which the paper presents as substantial agreement. The article does not provide a full annotation manual, detailed adjudication rules, or a complete label inventory in the main text. Nor does it offer a formal ontology of entity types. The safest interpretation is therefore that FinRED combines broad automatic construction with targeted human cleanup, especially on the benchmark test partition.

4. Benchmarking and empirical difficulty

To establish a benchmark, the 2023 paper evaluates three joint entity-relation extraction systems that do not rely on an external named entity recognizer: SPN, TPLinker, and CasRel. Evaluation is reported at triplet level using precision, recall, and $F_1$ based on exact entity matching. On FinRED, the reported results are: SPN with precision 89.22%, recall 86.42%, and $F_1$ 87.80%; TPLinker with precision 76.59%, recall 63.45%, and $F_1$ 69.40%; and CasRel with precision 69.71%, recall 55.84%, and $F_1$ 62.01% (Sharma et al., 2023).

These values are contrasted with reference results on general-domain datasets. The same table reports SPN at 92.30% $F_1$ on FB-NYT and 91.80% on WebNLG, TPLinker at 92.00% on FB-NYT and 86.70% on WebNLG, and CasRel at 89.60% on FB-NYT and 93.40% on WebNLG. The authors summarize the degradation as an $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$0 drop of about 4% for SPN and about 25–30% for TPLinker and CasRel. Their interpretation is that strong generic RE architectures do not transfer cleanly to the financial domain.

The paper also diagnoses where the difficulty lies. For SPN, relation $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$1 is 88.37% while entity $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$2 is 96.36%. The authors interpret this gap as evidence that entity prediction is relatively strong and that the major bottleneck is incorrect relation classification. They report similar trends for TPLinker and CasRel. This makes the domain-mismatch claim more specific: the principal challenge is not merely locating mentions, but learning finance-specific relation semantics.

A structural analysis using SPN further partitions performance by the number of triplets per sentence and by overlap type. By triplet count, the reported $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$3 values are 80.98% for 1 triplet, 87.60% for 2 triplets, 89.09% for 3 triplets, and 92.06% for 4+ triplets. By overlap type, the $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$4 values are 83.36% for NEO (No Entity Overlap), 89.16% for EPO (Entity Pair Overlap), and 87.98% for SEO (Single Entity Overlap). The authors find it notable that performance improves as the number of triplets increases, and they suggest that overlapping structures provide stronger cues and allow comparatively strong entity prediction to support relation prediction.

5. FinRED as a financial LLM red-teaming framework

The 2026 work introduces FinRED in a different sense: not as a relation extraction dataset, but as an expert-guided benchmark generation and evaluation framework for financial LLM safety. Its starting point is the claim that existing safety benchmarks target general adversarial scenarios and therefore miss finance-specific risks such as regulatory compliance violations, fraud facilitation, consumer harm, and systemic trust erosion. FinRED is presented as an end-to-end framework rather than a static dataset: experts define a financial threat taxonomy and schemas, relevant context is retrieved from real financial documents, an LLM generates structured threat scenarios and converts them into behavior seeds, downstream attack methods operate on those seeds, and a finance-specific judge rubric scores the resulting model outputs (Kim et al., 18 Jun 2026).

The benchmark was built with 12 domain experts from the Financial Security Institute, a South Korean government-related financial-security organization. The paper states that these experts were involved from the earliest stages, including literature review, focus group interviews, discussion sessions, schema refinement, realism validation, and rubric design. The framework is organized around a two-level taxonomy. Level 1 contains five broad domains: $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$5 Cyber Threats, $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$6 Financial Crime, $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$7 Misinformation and Deception, $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$8 Consumer Rights Violation, and $$(\text{Pemex}, \text{product or material produced}, \text{petroleum})$$9 Financial ICT Compliance Evasion. These are decomposed into 26 Level-2 categories.

The released artifact contains 5,805 expert-validated seeds distributed across these five Level-1 domains and 26 Level-2 categories. The paper reports the Level-1 totals as 792 seeds for $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$0, 1,260 for $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$1, 1,503 for $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$2, 1,044 for $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$3, and 2,205 for $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$4. It also reports an overall average prompt length of 308.2 tokens. The distribution is explicitly not class-balanced; the authors say it reflects real-world coverage rather than a synthetic balancing scheme.

A significant design feature is the explicit mapping from international frameworks and standards into benchmark dimensions. The paper names FATF, BIS/BCBS, ISO/IEC 27001, NIST, OWASP, and the EU’s DORA as core sources. These are not treated as direct compliance checklists. Instead, their threat-modeling content is abstracted into reusable benchmark dimensions such as AML evasion, operational-resilience failure, consumer harm, and cybersecurity abuse. FinRED is also reported to be deployed in South Korea’s Financial Security Institute regulatory sandbox for generative AI security evaluation in real financial services.

6. Generation pipeline, rubric design, and validation

The 2026 FinRED pipeline begins with approximately 500 financial documents selected and reviewed by domain experts. These include regulatory frameworks, supervisory guidelines, audit reports, and risk-assessment reports. Documents are processed using a two-stage chunking pipeline based on the Unstructured library with Chipper, a layout-aware parser that outputs JSON with layout metadata. Structural chunking begins at titles or tables, merging elements under 200 characters into blocks up to 2,800 characters; any chunk longer than 1,200 characters is then re-segmented with a 150-character overlap. The authors describe this as a way to preserve semantic coherence and document hierarchy while respecting embedding limitations (Kim et al., 18 Jun 2026).

Retrieval is performed separately for each Level-2 category. Expert-crafted queries are written for every Level-2 subtype. Chunks are embedded using OpenAI embeddings and stored in a Chroma vector database, and for each subtype the system retrieves the top-8 semantically relevant chunks. The retrieved material is treated as an “inverted playbook”: documents describing what financial institutions must not do are transformed into prompts describing harmful behavior that a safe model should refuse.

Prompt generation is schema-driven. For each Level-1 category, the authors define JSON schemas partitioned into “Essential cues” and “Optional cues.” The LLM first fills the schema in JSON form, producing an intermediate structured scenario with actors, context, goals, motivation, methods, and intended outcomes. That structured scenario is then converted into a natural-language behavior seed. The paper highlights dynamic persona assignment and systematic variation of optional elements to increase diversity while preserving threat structure. Examples of schema field names mentioned in the paper include attackerProfile, targetTechnology, vulnerabilityHypothesis, and requesterPersona.

The paper validates generation quality through a blind expert study comparing three pipelines: $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$5 Context-Free Generation, $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$6 Context-Aware Direct Generation, and $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$7 the full schema-driven FinRED pipeline. About 90 prompts from each pipeline were sampled, giving 270 unique prompts, which were shuffled and blindly rated by the 12 Financial Security Institute experts on Financial Risk Alignment, Threat Plausibility, and Specificity / Actionability. The paper reports that $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$8 scored best across all three criteria. A paired operational ablation on the same 270-prompt subset used attack success rate as the main outcome. The reported average ASR values for $$(\text{Pemex}, \text{headquarters location}, \text{Mexico City})$$9, $$(\text{More Than}, \text{industry}, \text{finance})$$0, and $$(\text{More Than}, \text{industry}, \text{finance})$$1 are respectively 41.32%, 49.86%, and 58.05% on general-purpose sLMs; 52.47%, 61.73%, and 70.28% on finance-specific sLMs; and 28.91%, 36.58%, and 44.44% on API LLMs.

Evaluation in FinRED is performed with a finance-specific judge rubric containing five dimensions: Harmfulness, Persuasiveness, Refusal quality, Factualness, and Evasiveness. The aggregation rule is intentionally conservative: a response is labeled Unsafe if any rubric dimension is judged Unsafe. The paper argues that this is appropriate in regulated financial settings, where partial assistance and indirect hints can still create unacceptable risk. The rubric was compared with HarmBench on 130 prompt-response pairs sampled across all 26 Level-2 categories. Against expert majority-vote labels, HarmBench achieved 76.92% agreement while the FinRED Judge achieved 88.46%, for an improvement of 11.54 percentage points; error counts fell from 30 to 15, recall improved from 0.73 to 0.88, Cohen’s $$(\text{More Than}, \text{industry}, \text{finance})$$2 increased from 0.47 to 0.68, and critical false negatives dropped from 28 to 12.

The main attack evaluation spans Direct Request, GCG, AutoDAN, TAP, GPTFuzzer, and AutoDAN-Turbo across general-purpose sLMs, finance-specific sLMs, and API-based LLMs. The paper reports that vulnerability is concentrated in open-source and finance-specific small models, while major API models are comparatively robust. GPTFuzzer is identified as the strongest black-box attack overall, and direct requests already achieve nontrivial ASR, which the authors interpret as evidence that the seeds themselves are adversarially strong.

7. Significance, limitations, and interpretation

The two FinRED resources are linked by a common methodological thesis: finance is not well served by generic benchmarks. The 2023 dataset argues that general-domain relation extraction benchmarks do not adequately represent financial relation sets or textual realizations. The 2026 framework makes the parallel claim that general LLM safety benchmarks fail to capture the subtle, regulation-entangled harms that characterize financial misuse (Sharma et al., 2023, Kim et al., 18 Jun 2026).

Their limitations, however, are different. The 2023 FinRED is built through distant supervision and therefore inherits label noise, entity ambiguity, and incomplete documentation of the alignment pipeline. The paper does not provide a full relation list, per-class counts, or detailed annotation guidelines in the main text, and the resource is sentence-level rather than document-level, which is particularly consequential for earnings calls. The benchmark section is also concise, with little detail on model settings and no comparisons against domain-adapted pretrained LLMs.

The 2026 FinRED is limited less by weak supervision than by access control, expert scarcity, and dual-use risk. The framework is intentionally gated for qualified researchers because it contains realistic financial red-teaming prompts. The same 12 Financial Security Institute experts participated in both design and evaluation, although the paper reports source-concealed evaluation, randomized prompts, temporal separation of tasks, and majority-vote stabilization to reduce confirmation bias. The authors note that this is not as strong as using a fully independent evaluation panel.

Several misconceptions follow from the shared name. The 2023 FinRED is not a financial LLM safety benchmark; it is a relation extraction dataset focused on finance-domain triplets. Conversely, the 2026 FinRED is not merely a static prompt dataset; it is an applied generation-and-evaluation framework with a threat taxonomy, document retrieval layer, schema-driven prompt construction, and a finance-specific judge rubric. A further ambiguity is bibliographic: the 2026 paper is titled "FFinRED," yet the framework itself is described throughout as FinRED.

Taken together, the two works show how the same label has been attached to two distinct forms of finance-specific evaluation: structured information extraction and adversarial safety testing. This suggests that FinRED has become a marker for a broader methodological commitment—namely, that financial NLP and financial AI safety require benchmarks whose corpora, relation inventories, threat models, and evaluation criteria are native to the domain rather than inherited from general-purpose settings.