InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning

Abstract: Outcome-reward reinforcement learning (RL) has proven effective at improving the reasoning capabilities of LLMs. However, standard RL assigns credit only at the level of the final answer, penalizing entire reasoning traces when the outcome is incorrect and uniformly reinforcing all steps when it is correct. As a result, correct intermediate steps may be discouraged in failed traces, while spurious steps may be reinforced in successful ones. We refer to this failure mode as the problem of credit assignment. While a natural remedy is to train a process reward model, accurately optimizing such models to identify corrective reasoning steps remains challenging. We introduce Intervention Training (InT), a training paradigm in which the model performs fine-grained credit assignment on its own reasoning traces by proposing short, targeted corrections that steer trajectories toward higher reward. Using reference solutions commonly available in mathematical reasoning datasets and exploiting the fact that verifying a model-generated solution is easier than generating a correct one from scratch, the model identifies the first error in its reasoning and proposes a single-step intervention to redirect the trajectory toward the correct solution. We then apply supervised fine-tuning (SFT) to the on-policy rollout up to the point of error concatenated with the intervention, localizing error to the specific step that caused failure. We show that the resulting model serves as a far better initialization for RL training. After running InT and subsequent fine-tuning with RL, we improve accuracy by nearly 14% over a 4B-parameter base model on IMO-AnswerBench, outperforming larger open-source models such as gpt-oss-20b.

• The paper introduces Intervention Training (InT) that identifies and corrects the first error in a reasoning trace to improve credit assignment.
• It employs self-proposed, step-specific interventions instead of end-result rewards, achieving a 20-fold increase in correct outcomes.
• The approach enhances RL training dynamics, yielding robust performance on complex benchmarks like IMO and math reasoning tasks.

InT: Self-Proposed Interventions for Step-Level Credit Assignment in LLM Reasoning

Motivation and Credit Assignment Challenge in RL for LLMs

LLMs fine-tuned with RL on complex reasoning tasks, especially mathematics or multi-step problem solving, confront a fundamental challenge in credit assignment. In standard outcome-reward RL, the entire reasoning trajectory receives credit or blame solely based on the final answer, regardless of whether most steps were correct except for a single critical error. This leads to suppression of valid reasoning steps within otherwise failed solutions and reinforcement of spurious steps within lucky successes. As the complexity and length of unsolved problems grow—typified by mathematical olympiad benchmarks—the signal-to-noise ratio in RL vanishes, and training plateaus because incorrect trajectories dominate and advantages collapse.

Intervention Training (InT): Method Overview

The core contribution is Intervention Training (InT), a credit assignment paradigm leveraging an LLM’s ability to self-verify and propose targeted single-step interventions on its own failed reasoning trajectories. Instead of learning and optimizing expensive, unstable process reward models (PRMs), InT instructs the LLM to perform a "diff" between its own solution and a reference solution (human or model-generated), detect the first incorrect step, and propose a corrective step—the "intervention." This intervention, when substituted and used as a prefix for continued sampling, is empirically shown to enable many successful completions otherwise unreachable by the base model.

Figure 1: Intervention training (InT) scheme, in which the model identifies erroneous steps in its underperforming reasoning traces, proposes step-localized corrective interventions, and is fine-tuned to increase the likelihood of these corrections.

This process can be visualized as follows:

Figure 2: Concrete examples of reasoning failures by Qwen3-4B-Instruct and their step-local interventions that, once applied, directly lead to correct answers in subsequent rollouts.

Empirical Validation: Efficacy of Self-Proposed Interventions

Across a diverse suite of open-domain and mathematical reasoning benchmarks (e.g., IMO-AnswerBench, AMO-Bench, Apex Shortlist), the authors systematically:

• Use Qwen3-4B-Instruct to generate many failed trajectories on hard problems.
• For each trajectory, apply InT to detect the first mistake and propose an intervention.
• Empirically demonstrate that continuing the rollout from $$(\text{problem}, \text{prefix before error}, \text{intervention})$$ leads to at least a 20-fold increase in the rate of obtaining a correct solution over the original erroneous prefix. The result is that InT can recover non-trivial signal from trajectories that would provide zero learning signal to outcome-reward RL.

Supervised Patch and RL Initialization

The dataset of on-policy prefixes plus interventions is used to perform SFT (supervised fine-tuning). Key design choices include cloning the incorrect prefix (ensuring relevance of the intervention) but not the suffix (to preserve exploration). Only interventions demonstrated empirically to sometimes lead to correct rollouts are included.

This model is then used to initialize RL post-training, typically with a method like GRPO. The empirical result is that the RL process, starting from InT-patched weights, gains vastly improved coverage of difficult problems and significantly higher pass@$k$ and pass@1 metrics on both train and test benchmarks.

Figure 3: SFT with InT trajectories yields the highest base-model log-likelihood and outperforms other guidance strategies (e.g., reference solution cloning, self-reflection) on both train and test pass@$k$.

Training Dynamics and Comparative Results

Empirical results highlight several strong claims:

• InT-patched models have dramatically higher reward improvements and lower zero-advantage ratios during RL post-training on hard sets, compared to all RL and SFT+RL baselines.
• Across standardized benchmarks, the full InT+RL pipeline with a 4B model surpasses even larger open-source models (e.g., outperforming GPT-OSS-20B on IMO-AnswerBench).
• The interventions themselves are shown to be highly "on-policy" — their tokens enjoy significantly higher likelihoods under the base model distribution than oracle or reference-solution traces. This preserves low-entropy, stable model initialization conducive to more effective RL.

  Figure 4: Pass@$k$ over RL training iterations demonstrates consistent and broad improvement from InT-initialized RL across the entire frontier compared to direct RL or reference solution SFT+RL.

Qualitative and Mechanistic Insights

• Interventions are typically concise—a small number of tokens compared to the original trajectory—yet induce solution paths that diverge sharply from the original (erroneous) rollout, opening up otherwise inaccessible branches in the solution search tree.
• The instruction-following ability of the LLM is crucial—models primarily tuned for reasoning but lacking instruction obedience generate lower-quality or unusable interventions.
• Intervention quality scales with model capacity and instruction-following fidelity, but even self-improvement without larger teacher models is effective.

  Figure 5: Visualization of diverging solution paths between InT and non-interventional rollouts, underscoring step-level intervention as a branching point toward success.

Theoretical and Practical Implications

The InT methodology delivers a practical solution to LLM step-level credit assignment on complex, long-horizon reasoning domains, circumventing the instability and compute overhead associated with PRMs. The self-proposed, single-step intervention mechanism enables credit localization and reallocation in settings where successful trajectories are rare and exogenous and synthetic feedback (hints, critiques) have limited utility.

Practically, this paradigm enables high RL sample efficiency on new and harder benchmarks, supports continual model self-improvement, and provides a framework compatible with both online RL and offline RL from suboptimal data.

Theoretically, InT advances the understanding of how the asymmetry in verification versus generation in LLMs can be exploited to amortize value estimation and policy optimization into a single, tractable improvement mechanism.

Future Directions

• Fully reference-free self-improvement: By directly strengthening verification abilities through standalone verification data, models can potentially eliminate reliance on human or external solutions, enabling autonomous continual self-improvement.
• Continual and memory-augmented settings: Adapting InT to multi-turn, summarized, or memory-compressed contexts to trace credit over persistent representations remains open and important.

Conclusion

InT sets a new standard for step-level credit assignment in RL for LLM reasoning. Utilizing self-proposed, reference-solution-guided corrections for failed trajectories and leveraging SFT+RL, it demonstrably surpasses previous approaches on challenging mathematical and logical reasoning tasks. Its simplicity, stability, and empirical efficacy make it a pivotal technique for enabling further advances in autonomous model self-improvement and robust open-domain reasoning in LLMs.