EvoCoT: Self-Evolving Curriculum for RLVR

• The paper introduces a self-evolving curriculum that uses answer-conditioned chain-of-thought verification to overcome sparse reward challenges in reinforcement learning.
• It employs a two-stage process by first generating verified reasoning paths from (Q, A) pairs and then progressively shortening the chain-of-thought to widen exploration.
• EvoCoT demonstrates significant performance gains on benchmarks like GSM8K and MATH compared to GRPO and SFT, transforming unsolved hard problems into effective training signals.

Searching arXiv for the specified paper and closely related work on evolutionary or self-evolving chain-of-thought methods. EvoCoT is a self-evolving curriculum learning framework for reinforcement learning with verifiable rewards (RLVR) for post-training LLMs on mathematical reasoning under sparse outcome supervision. It is designed to address the exploration bottleneck that arises when RLVR rewards only the final answer and rollout accuracy on hard problems is too low for the model to discover correct reasoning trajectories with meaningful frequency. The framework operates by first constraining the exploration space through self-generated and verified chain-of-thought (CoT) trajectories, then expanding that space in a controlled manner by progressively shortening the provided reasoning context. In the formulation introduced in "EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning" (Liu et al., 11 Aug 2025), this enables learning from initially unsolved hard problems without external CoT annotations or stronger teacher models.

1. Problem formulation and motivation

EvoCoT targets reinforcement learning with verifiable rewards in settings where supervision is available only as question–answer pairs $(Q, A)$, and reward is determined by whether the final answer is correct. The paper characterizes the central failure mode as an exploration bottleneck: on difficult math tasks, a model’s rollout accuracy is so low that it rarely encounters a valid solution path, leaving RL with extremely sparse training signal (Liu et al., 11 Aug 2025).

Within this setting, RLVR is limited by the size of the implicit reasoning search space. If the model cannot already reach a correct trajectory with nontrivial probability, the final-answer reward provides little guidance for improving intermediate reasoning. The paper contrasts this with existing approaches that either rely on stronger LLMs for distillation or filter out difficult problems. In the authors’ framing, both strategies weaken the scalability or the reasoning benefit of exploration because they either introduce teacher dependence or discard unsolved hard examples rather than turning them into training signal (Liu et al., 11 Aug 2025).

EvoCoT addresses this by attempting to learn directly from problems that the model initially fails to solve. Its core idea is to constrain exploration first, using self-generated reasoning paths that are answer-consistent, and then gradually expand the model’s freedom to explore by removing parts of those paths over training. This suggests a curriculum over partial reasoning traces rather than a curriculum over difficulty-filtered examples.

2. Two-stage self-evolving framework

The method is organized as two nested stages that repeat iteratively. The first stage constructs candidate reasoning supervision from answer supervision. The second stage converts those verified trajectories into a step-wise curriculum for RL training (Liu et al., 11 Aug 2025).

Stage 1: answer-guided reasoning path self-generation

Given only $(Q, A)$, the current LLM generates a chain-of-thought $C$ conditioned on both the question and the final answer:

$(Q, A) \xrightarrow[\text{LLM}]{} C$

The underlying assumption stated in the paper is that reasoning is easier when the final answer is provided. EvoCoT then verifies the generated trajectory by checking whether the model can recover the correct answer when conditioned on the generated CoT:

$$(Q, C) \xrightarrow[\text{LLM}]{} \hat{A}, \quad \text{keep } C \text{ iff } \hat{A} = A$$

Only CoTs that pass this consistency test are retained. These are then split into step-wise reasoning units:

$$(Q, C, \hat{A}) \xrightarrow[\text{split}]{} (Q, \hat{C} = \{c_1, c_2, ..., c_n\}, \hat{A}), \quad \hat{A} = A$$

The paper explicitly presents this stage as creating reasoning supervision from outcome supervision, without human CoT labels and without stronger teacher models (Liu et al., 11 Aug 2025).

Stage 2: step-wise curriculum learning

Once verified CoTs are available, EvoCoT constructs a curriculum by progressively removing reasoning steps from the tail. Training begins with the full CoT and then shortens it step by step:

$$\begin{aligned} &(Q, c_1, c_2, \dots, c_{n-1}, c_n) \to (C^*, A^*) \ &(Q, c_1, c_2, \dots, c_{n-1}) \to (C^*, A^*) \end{aligned}$$

$\vdots$

$$\begin{aligned} &(Q, c_1) \to (C^*, A^*) \ &(Q) \to (C^*, A^*) \end{aligned}$$

Here $C^*, A^*$ denote the model’s generated reasoning chain and answer. The authors’ intuition is that longer CoT guidance makes the problem easier, while shorter guidance expands the exploration space. The curriculum therefore moves from strongly constrained reasoning toward open-ended reasoning in a controlled manner under sparse rewards (Liu et al., 11 Aug 2025).

The framework is self-evolving because the full generate–verify–train cycle is repeated across iterations. As the model improves, its Stage 1 reasoning trajectories can also improve, enabling subsequent rounds of curriculum construction.

3. Algorithmic structure and implementation details

The paper provides pseudocode for the full method. At a high level, each self-evolving iteration performs the following operations: generate CoTs from $(Q, A)$0, verify them by answer recovery, split retained CoTs into steps, train on partial-CoT curricula from full length to zero length, and repeat for multiple iterations (Liu et al., 11 Aug 2025).

The practical meaning of this loop is concise but important. First, the model generates candidate explanations conditioned on the known answer. Second, only trajectories that can reproduce the answer are kept. Third, those trajectories are converted into a graded sequence of partially observed reasoning prefixes. Fourth, RL fine-tuning is carried out on that sequence, with reward based on whether the model’s answer matches the known answer. This structure is what allows EvoCoT to transform unsolved hard examples into usable RL signal.

The paper emphasizes that EvoCoT is orthogonal to prior RL training paradigms and can be inserted after a base RLVR stage. In the reported implementation, Stage 2 uses GRPO as the advantage estimator, but the framework is described as compatible with RL-based fine-tuning pipelines rather than tied to a single backend (Liu et al., 11 Aug 2025).

Training setup

EvoCoT is applied to multiple open model families:

Model families used	Models
Qwen	Qwen2.5-7B
Llama	Llama3.1-8B
DeepSeek	DeepSeek-Math-7B; DeepSeek-R1-Distill-Qwen-1.5B

Training uses the GSM8K and MATH training sets. For Stage 1, the method selects problems that the model fails to solve in all 8 rollouts. For each such unsolved problem, it samples 8 reasoning paths with temperature 1.0 (Liu et al., 11 Aug 2025).

The reported Stage 2 hyperparameters are: GRPO as the advantage estimator, learning rate $(Q, A)$1, train batch size 32, mini-batch size 32, prompt length max 3000, response length max 5192, samples per problem 8, temperature 1.0, KL loss enabled Yes, KL loss coefficient 0.0001, shuffle dataset No, and micro batch size 1. Training is done on 8×A100 (40GB) GPUs (Liu et al., 11 Aug 2025).

Evaluation protocol

Evaluation uses GSM8K test, MATH test, AIME 2024, AMC 2023, Minerva Math, and OlympiadBench. The metric is pass@k with $(Q, A)$2 in all experiments. Generation settings for evaluation are context length 8192, decoding temperature 0.6, and 8 sampled responses per model (Liu et al., 11 Aug 2025).

4. Empirical performance

The paper organizes its empirical claims around four research questions: whether EvoCoT solves previously unsolved training problems, whether it generalizes to unseen benchmarks, how it compares to GRPO and SFT, and whether self-evolution continues indefinitely (Liu et al., 11 Aug 2025).

Solving previously unsolved training problems

The method is explicitly designed to overcome the exploration bottleneck on unsolved training examples. Reported training-set improvements relative to GRPO include:

Model	Reported change
Qwen2.5-7B	Avg 84.6 → 89.1
R1-Qwen-1.5B	Avg 68.2 → 89.9
MATH for R1-Qwen-1.5B	+32.1

The paper also reports that EvoCoT largely preserves rollout success while reasoning trajectories become shorter. This indicates that performance does not collapse as step-wise guidance is reduced through the curriculum (Liu et al., 11 Aug 2025).

Generalization to unseen math benchmarks

The benchmark table reports average gains from 40.3 to 53.5 for Qwen2.5-7B and from 54.2 to 65.7 for R1-Qwen-1.5B. A specific highlight is that R1-Qwen-1.5B on OlympiadBench reaches 51.6, described as the best among the reported variants. The paper also states that EvoCoT is competitive with or better than SimpleRL, DeepScaleR, Open-Reasoner-Zero, and PRIME despite using only GSM8K and MATH training data (Liu et al., 11 Aug 2025).

Comparison with GRPO and self-training

The paper states that EvoCoT consistently outperforms both vanilla GRPO and SFT based on self-generated verified CoTs across all model families. Examples given in the benchmark table are:

Model	GRPO	SFT	EvoCoT
R1-Qwen-1.5B	63.6	55.3	65.7
Qwen2.5-7B	51.2	36.9	53.5

The authors interpret this as evidence that EvoCoT enables more effective self-exploration than either pure RL or self-training alone, especially for stronger base models (Liu et al., 11 Aug 2025). A plausible implication is that verified-CoT supervision by itself is insufficient unless it is coupled to the progressive relaxation of reasoning constraints.

Iterative improvement and saturation

Self-evolution does not continue indefinitely. The reported trend shows that gains typically plateau after 1–2 iterations. The paper provides the following examples: R1-Qwen-1.5B improves as $(Q, A)$3; Qwen2.5-7B as $(Q, A)$4; and Llama3.1-8B peaks early and then drops (Liu et al., 11 Aug 2025).

This indicates that iterative self-improvement is real but bounded. The saturation behavior is explicitly linked to Stage 1 trajectory quality and dataset limitations rather than presented as a universal failure of the curriculum mechanism.

5. Comparative position within the literature

EvoCoT is situated against several strands of work: distillation-based reasoning improvement, difficulty-filtering or curriculum methods, and RLVR baselines. The comparison table in the paper distinguishes EvoCoT as both distillation-free and unfiltered, in contrast to methods such as ReLIFT, TAPO, LUFFY, Guide-GRPO, SRFT, AdaRFT, RORL, and SEC (Liu et al., 11 Aug 2025).

This distinction matters because EvoCoT does not require stronger teachers and does not discard hard problems. In the authors’ formulation, its contribution is not merely another RL fine-tuning recipe, but a mechanism for converting previously unusable hard examples into trainable instances under sparse-reward conditions.

The term “EvoCoT” should not be conflated with other, similarly named or conceptually adjacent systems. "CoEvo: Continual Evolution of Symbolic Solutions Using LLMs" (Guo et al., 2024) does not define “EvoCoT” as a formal term; its actual method is CoEvo, which combines tree-based idea search, evolutionary operators, and a dynamic knowledge library for symbolic solution discovery. "CoTEvol: Self-Evolving Chain-of-Thoughts for Data Synthesis in Mathematical Reasoning" (Wang et al., 16 Apr 2026) is an evolutionary framework that treats each CoT as an individual in a population and improves those trajectories through selection, reflective global crossover, and uncertainty-guided local mutation. "CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific Reasoning" (Feng et al., 15 Oct 2025) focuses on scientific reasoning and develops an evolutionary CoT distillation pipeline with multi-thinker initialization, novelty-driven selection, reflective recombination, and mutation.

These related methods share a broad theme—evolution or iterative refinement over reasoning trajectories—but they address different regimes. CoEvo targets symbolic solution discovery with open-ended knowledge reuse (Guo et al., 2024). CoTEvol treats CoT synthesis as a population-based search over reasoning trajectories for data generation (Wang et al., 16 Apr 2026). CoT-Evo addresses scientific-domain CoT distillation under heterogeneous teacher failure modes (Feng et al., 15 Oct 2025). EvoCoT, by contrast, is specifically a self-evolving curriculum learning framework for RLVR that constrains and then gradually expands exploration on hard math problems (Liu et al., 11 Aug 2025).

6. Assumptions, limitations, and interpretation

The paper frames EvoCoT with three assumptions. First, given $(Q, A)$5, the model is more likely to produce a valid reasoning chain. Second, training with longer CoT guidance is easier than with shorter or no CoT. Third, iterative training improves the model enough to generate better CoTs in later rounds (Liu et al., 11 Aug 2025).

The limits of self-evolution are also specified. The paper identifies two main causes of saturation. One is dataset answer errors: if some training answers are wrong, the model can be penalized even when its reasoning is correct. The other is failure to generate a valid CoT from $(Q, A)$6: some problems remain unsolved even with the answer provided, which prevents Stage 1 from producing usable trajectories. These failures are filtered out and define the saturation point of self-evolution (Liu et al., 11 Aug 2025).

The discussion further suggests that stronger base models benefit more because they are better at producing consistent self-generated CoTs in Stage 1. This suggests that EvoCoT’s gains depend materially on the initial model’s ability to exploit answer-conditioned reasoning, even though the framework is designed to help models escape sparse-reward dead zones (Liu et al., 11 Aug 2025).

A common misconception would be to interpret EvoCoT as external CoT distillation. The paper explicitly positions it otherwise: it improves reasoning without external chain-of-thought supervision and without stronger teacher models. Another potential misconception is to treat the method as a generic RL algorithm. More precisely, it is a curriculum framework layered on top of RLVR training, with GRPO used in the reported experiments but not treated as essential to the concept itself (Liu et al., 11 Aug 2025).

7. Significance

EvoCoT’s significance lies in showing that hard, previously unsolved training examples need not be excluded from RLVR simply because the reward is sparse. By self-generating and verifying reasoning trajectories, then training over a sequence of progressively shorter partial-CoT conditions, the framework turns outcome supervision into curriculum-structured exploration signal (Liu et al., 11 Aug 2025).

In methodological terms, EvoCoT contributes a particular answer to a recurrent problem in reasoning-oriented RL: when final-answer reward is too sparse for direct exploration, structure can be introduced through intermediate reasoning traces without requiring human CoT labels or stronger teachers. This is not equivalent to ordinary self-training, because the critical mechanism is the controlled expansion of the exploration space as guidance is removed. It is also not equivalent to difficulty filtering, because the hardest initially unsolved examples remain central to the method rather than being discarded.

Within the broader landscape of self-improving reasoning systems, EvoCoT exemplifies a curriculum-based route to self-evolving exploration, distinct from the population-based evolutionary search used in CoTEvol (Wang et al., 16 Apr 2026), the knowledge-library-driven continual search of CoEvo (Guo et al., 2024), and the multi-teacher evolutionary distillation of CoT-Evo (Feng et al., 15 Oct 2025). Its specific contribution is to adapt self-generated reasoning structure to the RLVR setting, where the principal obstacle is not a lack of candidate data per se, but the inability of sparse-reward rollouts to discover correct trajectories often enough for learning to proceed.