CaP: Cross-Reasoning Refinement for LLMs

• The paper introduces CaP as a cross-reasoning refinement framework where a natural language CoT draft is transformed into a tool-augmented PoT response to repair errors.
• CaP leverages a two-stage training process, combining supervised fine-tuning with DPO-style preference optimization, to encourage genuine corrective behavior.
• The framework enhances inference reliability by allocating compute across initial CoT generation and subsequent tool-aided PoT refinement, addressing common pitfalls in same-format refinement.

Searching arXiv for the target paper and closely related refinement work. CaP, introduced in “Teaching LLMs to Refine with Tools” (Yu et al., 2024), is a cross-reasoning refinement framework for LLMs in which a chain-of-thought (CoT) draft is refined by a program-of-thought (PoT) response that can invoke external tools. The framework is motivated by a specific failure mode in prior refinement methods: when refinement remains within the same reasoning format, such as CoT-to-CoT or code-to-code, models often exhibit “non-correcting” behavior, learning to continue or imitate the earlier response pattern rather than actually repairing errors. CaP addresses this by changing both the reasoning substrate and the training signal: it uses tool-augmented PoT as the refinement target, and it relies on a two-stage procedure of supervised fine-tuning followed by DPO-style preference optimization.

1. Problem formulation and motivation

CaP is framed around a gap in LLM refinement research. Existing approaches allow models to improve responses from feedback, but they usually do so within the same reasoning format. The paper argues that this setup is often weak even when refinement is explicitly prompted, and that same-format supervised or preference-based refinement can still produce non-correcting behavior. In that regime, the model tends to reproduce the style or local structure of the prior answer rather than correcting the underlying reasoning error (Yu et al., 2024).

The proposed alternative is “cross-reasoning refinement,” defined precisely as refinement across distinct reasoning formats. In CaP, the draft is a CoT response expressed in natural language, while the refinement target is a PoT response that can compute, verify, and repair the earlier reasoning through external tools such as executable code. This shift is presented as both more realistic and more powerful. It is more realistic because external verification is a natural mechanism for checking reasoning; it is more powerful because the refinement stage is not constrained to remain in the same expressive or computational format as the initial attempt.

A central contrast in the paper is therefore not between refinement and no refinement, but between same-format refinement and tool-mediated cross-format refinement. CoT-to-CoT refinement is reported as much less reliable and prone to non-correction, whereas CoT-to-PoT refinement introduces a distinct corrective pathway. This distinction is one of the paper’s core conceptual contributions.

2. CaP as a cross-reasoning framework

CaP uses CoT as the draft and PoT as the refinement target. The key intuition is that PoT can exploit external tools to verify and repair reasoning originally expressed in natural language. Rather than requiring human-written step-by-step correction traces, the framework constructs training data automatically: teacher models generate CoT and PoT candidates, and critic models score those candidates for correctness. The critics may be reference-based, using the ground-truth answer, or reference-free, judging correctness directly from the question and candidate response (Yu et al., 2024).

The framework therefore has three operational components. First, a draft CoT solution is produced. Second, critic supervision is attached to that draft, including signals such as “The problem-solving process might be correct” or “might be wrong.” Third, the model generates a PoT response conditioned on the question, the earlier CoT attempt, and the critic signal. The output PoT is expected not merely to restate the answer, but to perform tool-assisted verification or correction.

This organization makes CaP different from ordinary self-refinement loops. The model is not trained to rewrite a flawed CoT into another CoT of the same kind. Instead, it learns a conditional mapping from a natural-language reasoning trajectory to a tool-augmented repair trajectory. In effect, the draft and the repair inhabit different reasoning spaces.

3. Training data and optimization objectives

CaP is trained in two stages. The first stage is supervised fine-tuning (SFT). The SFT data mixes multi-turn CaP instances and single-turn CoT instances. In the notation given in the paper, the CaP portion of the training set is denoted $D_{\text{CaP}}$, while the direct CoT portion is denoted $D_{\text{CoT}}$. The corresponding objective $L_{\text{SFT}}$ trains the model both to produce correct CoT answers from questions and, more importantly, to produce a positive PoT response conditioned on a question, a CoT draft, and a critic message (Yu et al., 2024).

The multi-turn CaP instances are built from “Best-of-N” selected training pairs constructed from both CoT and PoT samples. This gives the model exposure to two distinct behaviors: direct problem solving in CoT, and corrective refinement into PoT. The SFT stage therefore supplies the structural template for cross-reasoning refinement, but the paper repeatedly reports that SFT alone yields only weak or marginal refinement gains.

The second stage is preference optimization. Here the model is trained on a DPO-style contrastive objective over positive and negative PoT refinements, using a preference dataset denoted $D_{\text{CaP}_{\text{pref}}}$. The conditioning context $z$ is either $(q, y_{\text{cot}^{+}}, c^{+})$ or $(q, y_{\text{cot}^{-}}, c^{-})$, and the loss $L_{\text{DPO}}$ is designed to prefer $y_{\text{pot}^{+}}$ over $y_{\text{pot}^{-}}$. The paper further states that an added SFT term on the positive PoT sample stabilizes training and helps control length.

The training interpretation advanced by the paper is unambiguous: preference optimization is the stage that teaches the model to refine rather than imitate. SFT introduces the format and conditional dependency structure, but DPO-style preference learning is what makes corrective behavior emerge.

4. Inference-time strategies and compute allocation

CaP is also studied as an inference-time computation-allocation problem. The simplest baseline is greedy decoding with one CoT attempt followed by one PoT refinement. The paper then compares this baseline with Best-of-N (BoN) sampling and with a two-stage strategy called BoNBoN (Yu et al., 2024).

Strategy	Mechanism	Reported role
Greedy	One CoT attempt, then one PoT refinement	Baseline
BoN	Multiple CoT or PoT samples ranked by a critic	One-stage sample selection
BoNBoN	Half budget for multiple CoT drafts, then multiple PoT refinements from the best CoT	Two-stage compute allocation

In BoNBoN, half of the sample budget is used to generate multiple CoT attempts; the best CoT sample, as judged by a critic, is then used as the starting point for multiple PoT refinements, which are again ranked by a critic. The stated rationale is that refinement quality depends strongly on the quality of the intermediate CoT attempt, so compute should be distributed across both the draft-generation and refinement stages rather than concentrated in only one of them.

The reported result is that, under the same sample budget, CaP benefits especially from better CoT drafts and from distributing inference-time compute across both reasoning stages. The paper states that BoNBoN can further improve performance compared with one-stage BoN. This suggests that the refinement mechanism is not independent of draft quality; the intermediate CoT state is an important bottleneck.

5. Empirical behavior and comparative findings

The most important empirical claim is that effective CoT refinement appears only after preference optimization. SFT alone improves the model somewhat, but the improvement gap between the initial CoT attempt and the refined PoT output remains small. After DPO is applied, that gap increases sharply, and the model becomes capable of refining even strong off-policy CoT responses from a much larger teacher model. The paper highlights a concrete example: a smaller CaP model can refine CoT from Qwen2-72B-Instruct (Yu et al., 2024).

The paper also reports a negative result that clarifies the mechanism. When the same training recipe is used with CoT itself as the refinement target, the system either does not improve or remains unchanged, and preference optimization does not rescue that setting. The implication drawn in the paper is explicit: the gain is not just “more alignment” in general. It depends on the introduction of tools and on the cross-reasoning structure.

A second pattern is that refinement works best when the initial CoT is neither too weak nor too strong. Stronger CoT drafts usually yield better final PoT performance, but refinement becomes harder when the draft model is far stronger than the refiner. Even so, CaP is described as more robust than same-format refinement and as able to generalize across backbone models, including Qwen2-7B, Qwen2.5-7B, and Llama3-8B, with DPO consistently outperforming SFT.

These findings bear directly on a common misconception about iterative reasoning systems. The paper does not present CaP as a generic self-correction method that will fix arbitrary reasoning if given enough preference optimization. Its empirical picture is more conditional: correction depends on tool access, on reasoning-format asymmetry, on critic quality, and on the competence gap between draft and refiner.

6. Limitations, practical implications, and significance

The paper identifies several limitations. First, preliminary experiments suggest that refining across more than one additional attempt yields only marginal gains. Second, alternating refinement directions—for example, adding PoT-to-CoT refinement alongside CoT-to-PoT—can hurt both abilities. Third, the method depends on critic quality: critics are automated, but the paper states that more detailed error-aware critics would likely help. Finally, CaP’s ability to refine is tied to the model’s own reasoning competence; if the quality gap between the draft and the refiner is too large, refinement becomes difficult (Yu et al., 2024).

The practical implications are presented in two directions. One concerns reliability: tool-augmented refinement can make outputs more reliable and less sensitive to superficial perturbations while reducing the need for manual process supervision. The other concerns adaptive inference-time compute: the paper suggests a strategy of spending more effort on generating a good draft and then refining it with tools. These implications follow directly from the observed benefits of cross-reasoning refinement and from the BoNBoN results.

In the broader context of LLM refinement, CaP reframes the problem from iterative rewriting to genuinely corrective transformation. Its central claim is that same-format refinement is often brittle because it encourages imitation of prior reasoning traces, whereas external tools combined with preference optimization can convert refinement into a process of verification and repair. On that reading, CaP is less a prompt-engineering trick than a statement about representation: correction becomes substantially more plausible when the model is trained to move from natural-language reasoning into a tool-augmented reasoning format rather than remaining within the same discourse form.