FEVO-Train: Staged Financial Training Corpus

• FEVO-Train is a curated multi-source training corpus divided into three subsets for continued pre-training, structured fine-tuning, and reinforcement learning.
• It integrates financial texts with chain-of-thought logic and outcome-based rewards, addressing knowledge gaps and enhancing open-ended reasoning.
• Empirical results show that the staged FEVO-Train pipeline significantly boosts performance on benchmarks like Fin CPA and FinanceIQ.

FEVO-Train is the curated, multi-source training corpus that underpins FEVO (“Financial Evolution”), a three-stage enhancement framework for LLMs in the financial domain (Pang et al., 8 Jul 2025). It is not a single monolithic dataset, but a family of three stage-specific corpora—FEVO-Train-C, FEVO-Train-S, and FEVO-Train-R—aligned respectively with continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL) on top of Qwen2.5-32B. Its central purpose is to expand financial domain knowledge, distill structured long-form reasoning, and then integrate both via RL through datasets curated with frontier reasoning models and rule-based filtering.

1. Definition and position within FEVO

FEVO is a three-stage post-training pipeline. In the first stage, CPT expands the base model’s financial domain knowledge by exposing it to large-scale financial text, including textbooks, exam questions, news, and corpora, producing FEVO-C32B. In the second stage, SFT instills structured, elaborate chain-of-thought reasoning patterns—including planning, reflection, and backtracking—producing FEVO-S32B. In the third stage, RL integrates the expanded financial knowledge with the learned structured reasoning using a VAPO-based PPO variant under an outcome-based reward scheme, producing FEVO-R32B (Pang et al., 8 Jul 2025).

Within that pipeline, FEVO-Train is the entire curated training corpus. Its three subsets are designed to address three distinct deficiencies: the knowledge gap, the reasoning-structure gap, and the knowledge–reasoning integration gap. The paper’s formulation is explicit that generic pretraining spreads capacity across many domains, so financial recall is weak and structured reasoning over financial knowledge is not well-internalized. FEVO-Train is therefore specialized for financial terminology, regulations, accounting standards, tax treatments, valuation rules, and multi-step numerical reasoning.

A common misconception is that FEVO-Train is a single dataset. In the FEVO framework, it is instead a staged data system whose structure mirrors the training schedule itself. This suggests that FEVO-Train is best understood as an engineered curriculum rather than as a conventional benchmark corpus.

2. Corpus architecture and source composition

The three FEVO-Train subsets differ in size, function, and admissible data forms. Their aggregate design is summarized below (Pang et al., 8 Jul 2025).

Subset	Size	Primary role
FEVO-Train-C	about 188M tokens	CPT for financial knowledge expansion
FEVO-Train-S	291K entries	SFT for structured long CoT distillation
FEVO-Train-R	32K items	RL for open-ended, verifiable reasoning

FEVO-Train-C draws from both finance and general-domain corpora. Its finance components are Acc Learn (329,419 tokens), FinCorpus QA (23,283,970 tokens), and IndustryCorpus Finance (24M tokens). Its general components are InfiMM-WebMath-Edu-zh (47M tokens), SkyPile-150B (47M tokens), and Dolma (47M tokens). The resulting CPT corpus is reported as roughly 188M tokens after processing.

FEVO-Train-S is assembled from Duxiaoman FinQA (10K items), Fin CPA Agent-instruct (1,121 items), CFLUE (20K items), and the applied mathematics portion of Chinese-DeepSeek-R1-Distill-data-110K, with 30K items used and quality score $=10$. After structured CoT generation and aggressive filtering, the final SFT dataset contains 291K entries.

FEVO-Train-R is built from Fin CPA (215 items), Duxiaoman Fill-in-blanks (23K items), Dianjin Fill-in-blanks (2K items), and AM-Math-Difficulty-RL (4K items), with FEVO using 400 easy, 600 medium, and 3000 hard samples from the last source. After conversion and filtering, the RL corpus contains 32K high-quality open-ended questions.

The source mixture is deliberate. Finance-heavy data supplies domain specificity, while general math, code, Chinese, and English corpora preserve general capabilities and output stability.

3. Construction pipeline and quality control

Across all stages, FEVO-Train is subjected to common preprocessing and filtering operations (Pang et al., 8 Jul 2025). These include cleaning and format removal, hyperlink elimination, image/table filtering, deduplication, short-entry elimination where relevant, and language/domain normalization. The deduplication procedure is described as using “a combination of LSM and MinHash,” with $n$-gram set to 13.

The SFT and RL subsets add stricter task-level quality control. In FEVO-Train-S, entries are removed when the generated answer does not match the provided reference answer. GPT-4o is used as a judge to determine whether the answer matches the problem and whether the response is a “fluke response” without a logical reasoning path. The pipeline also verifies solvability, removes prompts containing images, tables, or hyperlinks, and excludes prompts with multiple sub-questions so that each retained sample corresponds to a single well-defined task.

In FEVO-Train-R, filtering targets clarity, verifiability, solvability, and completeness. Each prompt must convey a single clear request, have a reference answer that can be robustly compared with model outputs, admit a valid solution, and include all required contextual information. The paper explicitly allows domain-knowledge assumptions such as the content of relevant laws or standard industry practice, but excludes missing problem-specific information.

The overall effect of these filters is not merely denoising. They operationalize what FEVO counts as trainable supervision at each stage: dense financial signal for CPT, logically organized traces for SFT, and outcome-verifiable open-ended tasks for RL.

4. FEVO-Train-C: continued pre-training and financial knowledge expansion

FEVO-Train-C is designed to improve financial domain recall and what the paper calls “keyword expansion capability” (Pang et al., 8 Jul 2025). It is used with a standard causal language modeling objective:

$$\min_\theta \; \mathbb{E}_{(x_1,\dots,x_T)\sim \text{FEVO-Train-C}} \left[- \sum_{t=1}^T \log p_\theta(x_t \mid x_{<t}) \right].$$

Its distinctive feature is the incorporation of frontier-model-generated reasoning expansions into the CPT corpus. From FinCorpus QA, which contains multiple-choice questions with answers, each question is processed with DeepSeek-R1 or another frontier reasoning model to convert the correct answer choice into a detailed reasoning response that thoroughly explains the steps taken to arrive at the solution. After deduplication, hyperlink removal, and short-entry elimination, these reasoning trajectories are appended to the CPT data.

As a result, FEVO-Train-C includes raw financial text, frontier-model-generated solution explanations over financial exam content, and general-domain material. The inclusion of general-domain data is explicitly justified as a way to maintain stable, coherent outputs and to prevent the model from degrading into a “rigid answering bot for financial test questions.”

The CPT training configuration is also specified: learning rate $3\times 10^{-4}$, warmup 0.02, train batch size 4, split size 2048, and 1 epoch, producing FEVO-C32B. Empirically, the paper reports that CPT-only training materially improves some finance-specific performance—most notably Fin CPA—while some general scores decline, which motivates the mixed-domain composition of FEVO-Train-C.

5. FEVO-Train-S: structured reasoning distillation

FEVO-Train-S is an instruction–response corpus whose responses are explicitly structured long CoTs with five labeled sections: Plan, Reasoning, Reflection, Backtracking, and Answer (Pang et al., 8 Jul 2025). The paper defines these sections as follows: Plan is an initial plan drafted to tackle the problem; Reasoning is a comprehensive thinking process to solve the problem step by step; Reflection is a review of the reasoning steps in the previous stage, checking for potential errors and potentially overturning existing thought process; Backtracking is an overall evaluation of all previous stages, utilizing potentially different approaches for solving attempt; and Answer is the final answer based upon all previous thought process.

The data is generated in two modes. In the first, DeepSeek-R1 produces a full structured CoT in a single response. In the second, Qwen2.5-72B-Instruct is used in a multi-round session to generate Plan, then Reasoning conditioned on the Plan, then Reflection, then Backtracking, and finally Answer; the parts are then concatenated into a single structured response.

The SFT objective is standard autoregressive supervised fine-tuning:

$$\min_\theta \; \mathbb{E}_{(q,y)\sim \text{FEVO-Train-S}} \left[- \log p_\theta(y \mid q)\right].$$

The resulting training configuration is learning rate $5\times 10^{-5}$, warmup 0.02, train batch size 1, cutoff length 8192 tokens, and 1 epoch, producing FEVO-S32B. The paper reports that, during preliminary experiments, fine-tuned models at this stage already exhibit well-structured, logic CoTs when responding to questions.

Conceptually, FEVO-Train-S does more than teach answer production. It codifies a particular reasoning style in which solution planning, stepwise derivation, self-checking, and alternative-attempt evaluation are all first-class textual objects.

6. FEVO-Train-R: open-ended RL data, reward design, and VAPO coupling

FEVO-Train-R is built for outcome-based RL and is explicitly engineered to reduce reward hacking (Pang et al., 8 Jul 2025). The underlying problem is that many raw financial QA sources are single- or multiple-choice; with a limited answer space, an LLM might guess the correct answer without understanding how to solve the problem. To address this, DeepSeek-R1 is used to rewrite choice-based or fill-in items into open-ended form. The rewritten prompt retains the full problem context, while an answer option is embedded into the text of a new question or removed entirely so that the model must produce the correct value or explanation rather than a choice index. Multiple open-ended questions can be synthesized from one MCQ by using different answer choices as the basis for separate variants.

RL on FEVO-Train-R uses format, accuracy, and language consistency rewards. Models are instructed to produce a thinking process inside > ... and a final answer inside \boxed{...}. If the format is incorrect, no correctness check is performed. The reward is

$$R_{\text{format+accuracy}} = \begin{cases} -2.0 & \text{if format incorrect} \ -1.5 & \text{if format correct, answer incorrect} \ 2.0 & \text{if format and answer both correct.} \end{cases}$$

To enforce monolingual responses in a bilingual corpus, the language consistency term is

$$R_{\text{language}} = \begin{cases} -0.5 & \text{if } p_{cn} < k \ \cap\ p_{en} < k \ 0.0 & \text{if } p_{cn} \geq k \ \cup\ p_{en} \geq k, \end{cases}$$

with threshold $k=0.8$. The total reward is

$$R = R_{\text{format+accuracy}} + R_{\text{language}},$$

and the RL objective is

$n$0

The optimization algorithm is a modified VAPO implementation in the veRL framework. The PPO term is

$n$1

the positive LM loss over correct responses $n$2 is

$n$3

and the combined objective is

$n$4

Advantages are computed with length-adaptive GAE:

$n$5

$n$6

with $n$7 in the experiments.

The RL rollout protocol uses 16 responses per prompt and 128 prompts per batch. Because naive batching produces many all-correct or all-wrong prompts, the paper introduces balanced batching. It maintains an accuracy record per question, defines hard questions with $n$8 and easy questions with $n$9, and caps them at $$\min_\theta \; \mathbb{E}_{(x_1,\dots,x_T)\sim \text{FEVO-Train-C}} \left[- \sum_{t=1}^T \log p_\theta(x_t \mid x_{<t}) \right].$$0 rollout batch size and $$\min_\theta \; \mathbb{E}_{(x_1,\dots,x_T)\sim \text{FEVO-Train-C}} \left[- \sum_{t=1}^T \log p_\theta(x_t \mid x_{<t}) \right].$$1 rollout batch size respectively. The remainder are medium-difficulty questions. On FEVO-Train-R, this produces about 20% faster training than default dynamic sampling with similar accuracy evolution.

7. Empirical significance, control comparisons, and limitations

The empirical rationale for FEVO-Train is strongest in the comparison between the full pipeline and an RL-only control, FEVO-R32B-0, which starts from Qwen2.5-32B-Instruct and applies RL without CPT or SFT (Pang et al., 8 Jul 2025). Across seven benchmarks, the average scores are 74.25 for Qwen2.5-32B-Instruct, 67.68 for FEVO-C32B, 76.65 for FEVO-S32B, 77.83 for FEVO-R32B-0, and 85.01 for FEVO-R32B. The paper characterizes FEVO-R32B as achieving state-of-the-art performance on five financial benchmarks.

The benchmark-level results are the clearest indicator of what FEVO-Train contributes. On Fin CPA, FEVO-R32B reaches 73.60, versus 63.04 for FEVO-R32B-0, 58.4 for FEVO-S32B, and 57.0 for FEVO-C32B. On Fin CCR, FEVO-R32B reaches 88.20. On FinanceIQ, it reaches 86.14, exceeding the 80.24 reported for Qwen2.5-72B-Instruct. On CFLUE, it reaches 85.2, second to Dianjin-R1-32B at 86.75. On FIN-EVA, it reaches 85.66. On OpenFinData, it reaches 87.70. On Math 500, it reaches 88.6, which the paper describes as competitive with dedicated reasoning models.

These results support the paper’s specific claim that there is a marked performance gap between R32B-0 and R32B, validating the effectiveness of financial domain knowledge expansion and structured, logical reasoning distillation. In FEVO terms, that gap is attributable precisely to FEVO-Train-C and FEVO-Train-S.

Several limitations are also implicit. Coverage is concentrated on CPA/accountant exams, Chinese regulatory and exam content, and Chinese financial news, so other financial subdomains and languages may be underrepresented. FEVO-Train is almost entirely text-based and makes minimal use of structured numerical series, order book data, or time-series features beyond exam-style math. Because CoT traces and open-ended conversions rely on DeepSeek-R1 and other frontier models, some noise may survive despite verification and judging. Reward granularity remains coarse, centered on correctness, format, and language rather than fine-grained process supervision. The future directions stated for the framework include better RL algorithms, more challenging financial tasks such as graph generation and legally-compliant document generation, and better reward models that provide more finely-grained rewards than current rule-based implementations.

In aggregate, FEVO-Train functions as the data backbone of a staged specialization strategy: finance-focused CPT for knowledge expansion, structured SFT for reasoning distillation, and open-ended RL for integrating the two. A plausible implication is that its design principles—frontier-model expansion, explicit long-CoT supervision, and MCQ-to-open-ended conversion for RL—are not confined to finance, but form a reusable recipe for domain-specific reasoning training.