ConDiFi: Financial Reasoning Benchmark

• ConDiFi is a finance-domain benchmark that evaluates both divergent scenario-generation and convergent timeline selection in LLMs.
• It features 607 prompts for branching macro-financial scenarios and 990 adversarial MCQs demanding precise causal and chronological reasoning.
• The benchmark emphasizes combining creative foresight with disciplined decision-making for safe, strategic deployment in financial markets.

ConDiFi, expanded in the source paper as “Convergent-Divergent for Financial Reasoning,” is a finance-domain benchmark for LLMs that jointly evaluates two complementary reasoning modes: divergent thinking, defined as the generation of multiple plausible and novel futures under uncertainty, and convergent thinking, defined as selecting the single best answer by integrating clues, constraints, and causal order. It was introduced to address a perceived gap in prevailing evaluation practice, where standard benchmarks emphasize factual accuracy, general multi-hop reasoning, or narrow step-by-step deduction, while professional finance requires both scenario generation and disciplined judgment. The benchmark comprises 607 macro-financial prompts for divergent reasoning and 990 adversarial multiple-choice questions for convergent reasoning, and it is positioned as a tool for assessing reasoning capabilities relevant to the safe and strategic deployment of LLMs in finance (Bok et al., 24 Jul 2025).

1. Conceptual scope and motivation

ConDiFi is motivated by the claim that finance is not adequately represented by benchmarks whose success criteria reduce to exact-match correctness on a single answer. The benchmark’s authors contrast their setting with general reasoning suites such as MMLU, ARC, OpenBookQA, GPQA, HotpotQA, 2WikiMultihopQA, BRIGHT, and MASTERMINDEVAL, which largely stress convergent reasoning, and with creativity-oriented tasks such as AUT and RAT, which they regard as weakly grounded for professional financial decision-making (Bok et al., 24 Jul 2025).

The benchmark therefore separates two cognitive regimes that finance often requires simultaneously. Divergent reasoning is framed as branching scenario construction: the model must imagine non-obvious but plausible future paths, including second- and third-order effects. Convergent reasoning is framed as timeline selection: the model must choose the single correct event sequence among subtly wrong alternatives. This separation is central to the benchmark’s design because analysts, portfolio managers, and high-intent retail investors routinely need both strategic foresight and disciplined disambiguation.

A further design choice is temporal freshness. The benchmark is deliberately built from post-May 2025 sources, with convergent items tied to NYSE-related scenarios, in order to reduce pretraining contamination and make performance depend more on reasoning than on memorized facts. This suggests that ConDiFi is intended not merely as another finance QA set, but as a stress test for reasoning under recent, domain-specific uncertainty.

Component	Size	Purpose
Divergent reasoning dataset	607 prompts	Branching macro-financial scenario generation
Convergent reasoning dataset	990 MCQs	Adversarial timeline selection

2. Divergent reasoning benchmark

The divergent component contains 607 scenarios, each described as a medium-length summary of a real-world event or development dated 1 May 2025 or later and kept under 500 words. The scenarios span economics, finance, geopolitics, and politics, with emphasis on developments that professional investors would regard as decision-relevant. The required output is not unconstrained prose but a structured tree of alternative future paths, with nodes representing future events and branches representing uncertainty-aware evolution (Bok et al., 24 Jul 2025).

Its construction is described as a multi-stage pipeline. Real financial scenarios are curated from credible post-cutoff sources; a model is prompted to generate a branching timeline; GPT-4o evaluates the timeline; a structural richness score is computed from the tree shape; and results are aggregated across 14 models, yielding 9,380 samples in total. The benchmark therefore mixes LLM-as-judge evaluation with a graph-structural metric, rather than relying on one scalar derived from a single rubric.

Five dimensions are evaluated. Plausibility measures whether the timeline obeys economic, political, and financial cause-effect logic. Novelty captures originality, especially through second- and third-order effects. In the reported rubric, Actionable is defined as assessing the level of detail in each node and the overall tree structure, while Elaboration is defined as evaluating whether the timeline yields specific investment takeaways such as tickers, sectors, asset classes, or hedging triggers. Richness is an automated graph-based metric. The judge rubric specifies that a score of about 5 corresponds to an average sell-side analyst note, that scores of 9–10 are reserved for the top 5% of timelines, that generic filler phrases and unsupported numbers are penalized, and that any timeline with logical impossibilities or temporal inconsistencies is capped at 3 on all dimensions (Bok et al., 24 Jul 2025).

The paper formalizes Richness by parsing a response as a directed tree and normalizing four structural quantities: average branching factor $b$, maximum path length $d_{\max}$, mean path length $\bar{d}$, and breadth $p$ measured as the number of leaf paths. The normalized components are reported as

$s_b = \min\left(\frac{b}{3}, 1.0\right),$

$s_d = \min\left(\frac{d_{\max}}{10}, 1.0\right),$

$$s_{\bar d} = \min\left(\frac{\bar d}{7}, 1.0\right),$$

$$s_p = \min\left(\frac{\log(1+p)}{\log(101)}, 1.0\right),$$

with the final score

$$\text{Richness} = \min\left(10,\ \max\left(1,\ 10 \times \frac{1}{4}(s_b + s_{\bar d} + s_d + s_p)\right)\right).$$

The stated purpose of this construction is to capture breadth of imagination, depth of causal chains, and explicit uncertainty modeling.

3. Convergent reasoning benchmark

The convergent component contains 990 adversarial multiple-choice questions. Each question is based on a real NYSE-listed company and a scenario derived from financial news from 2025 onward. The answer choices are not atomic labels but event timelines, and only one option is correct. A valid choice must satisfy factor alignment, temporal coherence, and logical entailment; the distractors are designed to fail in subtle, finance-specific ways rather than by obvious contradiction (Bok et al., 24 Jul 2025).

Question generation combines textbook factors relevant to a firm with recent news, after which GPT-4o is prompted to generate items through six adversarial pipelines. These are historical $\beta$-swap, numeric trip-wire, policy game, cross-section confuser, reg-legal trap, and adversarial self-play. Each pipeline targets a distinct failure mode, such as causal misunderstanding, small quantitative mistakes, procedural chronology errors, or confusion between firms with opposite exposures. The benchmark is then hardened by two rounds of refinement inspired by self-reflection workflows: if the generator can answer an item too easily, the prompt is rewritten to make the example less obvious.

Evaluation is summarized by the Convergent Correctness Score:

$d_{\max}$0

with $d_{\max}$1. In operational terms, CCS is exact-match accuracy over the adversarial MCQs. This metric is simple, but the dataset construction makes it demanding: the wrong options often remain plausible unless the model tracks chronology, factor weighting, and firm-specific exposure with precision.

The convergent design is significant because it moves beyond ordinary financial multiple choice. Instead of asking for a static fact or a one-step inference, it tests whether a model can reconcile business logic, causal sequence, and sector-specific constraints under adversarially crafted near-miss answers.

4. Experimental protocol and empirical findings

ConDiFi evaluates 14 models: Cohere Command A, Cohere Command R, GPT-4o, o1, o1-mini, o4-mini, Mistral-Large-2411, Mistral-small-2503, Meta-Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, Llama-4-Maverick-17B-128E-Instruct-FP8, Llama-4-Scout-17B-16E-Instruct, DeepSeek-R1, and Phi-4. Divergent evaluation uses GPT-4o as judge; convergent evaluation uses a one-shot chain-of-thought prompt instructing models to identify the correct timeline (Bok et al., 24 Jul 2025).

On divergent reasoning, the strongest overall scores are reported for Cohere Command A at 8.04 and DeepSeek-R1 at 8.03, followed by Phi-4 at 7.76 and o1 at 7.70. Lower scores include GPT-4o at 6.92, Cohere Command R at 5.85, and Llama-3.1-8B-Instruct at 5.13. The paper’s interpretation is that fluency alone does not imply strong creative financial reasoning: GPT-4o is described as fluent and decent on plausibility, yet underperforms on Novelty and Actionable. By contrast, DeepSeek-R1 is reported as especially strong on Novelty, Elaboration, and Actionable, although its Richness is weaker than that of some peers. The benchmark narrative particularly emphasizes Cohere Command A as strong in generating actionable, investment-useful timelines.

Across the 9,380 divergent samples, the mean scores are approximately 7.81 for Plausibility, 6.50 for Novelty, 7.53 for Elaboration, 6.24 for Actionable, and 5.91 for Richness. A plausible implication is that current models are materially better at producing plausible and moderately elaborated narratives than at generating genuinely novel, structurally rich, and decision-ready financial scenarios.

The reported cross-dimension correlation structure reinforces that point. Plausibility correlates with Novelty at 0.706 and with Actionable at 0.576, while Novelty correlates with Elaboration at 0.581. By contrast, Richness is only weakly correlated with the other dimensions, ranging from -0.043 to 0.068. This suggests that structural branching is not reducible to surface plausibility or elaboration. The paper extends this analysis with Frobenius distances between model-specific correlation matrices and identifies DeepSeek-R1 as a behavioral outlier, LLaMA-family models as relatively similar, and Cohere Command A as distinctive in its correlation structure.

On convergent reasoning, the best results on the original set are reported for Llama-4-Scout at 0.9567 and Llama-4-Maverick at 0.9547, followed by Llama-3.3-70B-Instruct at 0.9406, GPT-4o at 0.9376, DeepSeek-R1 at 0.9355, and o1 at 0.9245. The weakest reported performers are Llama-3.1-8B-Instruct at 0.5389 and Cohere Command R at 0.4931. Averaged across the original and refined datasets, accuracy falls from 0.8545 on the original set to 0.8331 after the first refinement and to 0.6561 after the second, which the authors take as evidence that the refinement process meaningfully increases difficulty.

5. Interpretation, error taxonomy, and deployment relevance

A central argument of ConDiFi is that model quality in finance cannot be inferred from generic benchmark accuracy alone. The benchmark is designed to expose a gap between sounding plausible and being useful. In the divergent setting, the decisive dimensions are not only coherence but also novelty, actionability, and branching structure. In the convergent setting, the decisive factor is not only factual recall but the ability to weight subtle firm- and macro-specific signals correctly.

The error analysis focuses especially on Llama-4-Maverick, described as the best-performing model on the convergent set in that analysis. Using GPT-4o as an error analyst, the paper classifies mistakes into six categories: misinterpretation of scenario nuances and key information at 68%, incorrect prioritization or weighting of factors at 62%, overlooking specific critical details at 28%, bias toward optimism or positive outcomes at 18%, flawed logical or causal reasoning at 14%, and misunderstanding of task objective or evaluation criteria at 7% (Bok et al., 24 Jul 2025). These distributions indicate that many failures are not simple knowledge deficits. They arise from causal weighting, exposure matching, and over-optimistic reading of mixed signals.

The benchmark also stratifies item difficulty. It reports that 128 questions were answered incorrectly by all models and 189 were answered correctly by all models, and groups items into Hard, Moderate, and Easy tiers. A model scoring at most 55% is described as likely solving only the easy items; a score from 55% to 78% corresponds to solving easy and some moderate items; and scores above 78% indicate success on easy, moderate, and some hard items. This tiering supports the claim that ConDiFi is not merely adversarial in the colloquial sense, but calibrated across levels of financial reasoning difficulty.

The deployment relevance emphasized by the paper is therefore twofold. First, a model that is persuasive but not actionable may be unsuitable for decision support. Second, a model that can answer conventional finance questions may still fail at scenario planning, policy spillovers, or causal chronology. ConDiFi is intended to make those distinctions visible for applications such as investment-thesis generation, scenario exploration, and risk-sensitive strategic support.

6. Limitations and disambiguation

The source paper notes several limitations. The benchmark is U.S.-centric and focused on post-May-2025 equities. Some contamination from training overlap may still exist despite post-cutoff sourcing. Divergent scoring relies on GPT-4o-as-judge and may therefore inherit judge bias. The Richness metric is said to correlate only moderately with human ratings and may reward breadth over depth. Only 14 public models were evaluated; Claude and Gemini were not included. Responses were capped at 4096 tokens, and prompting was deliberately minimal, without tools, few-shot exemplars, or decoding sweeps (Bok et al., 24 Jul 2025). These caveats matter because they bound how broadly the reported rankings should be generalized.

The name “ConDiFi” is also potentially confusable with several unrelated arXiv terms. CoFinDiff is a controllable financial diffusion model for synthetic time-series generation based on conditional diffusion, Haar wavelet representations, and cross-attention; it addresses synthetic financial data generation rather than LLM reasoning evaluation (Tanaka et al., 6 Mar 2025). ParCFDFinder, integrated into Desbordante, concerns the discovery of conditional functional dependencies in relational data, not finance reasoning benchmarks (Kozhukov et al., 4 Jul 2026). CDFI denotes “Compression-Driven Network Design for Frame Interpolation,” a model-compression framework for video frame interpolation (Ding et al., 2021). UniCombine is a DiT/MMDiT-based framework for multi-conditional controllable image generation; it has been described as relevant only in the loose sense of conditional fusion within diffusion, not as a method named ConDiFi (Wang et al., 12 Mar 2025).

Within current arXiv usage, ConDiFi therefore most precisely denotes the finance benchmark for convergent and divergent reasoning in LLMs. Its distinguishing contribution is not generative modeling, dependency discovery, or vision compression, but a domain-specific evaluation framework that treats financial reasoning as the joint problem of imagining plausible futures and selecting the best justified path among adversarial alternatives.