---
title: Chain-of-Thought Supervised Fine-Tuning
url: https://www.emergentmind.com/topics/chain-of-thought-supervised-fine-tuning
type: topic
---

# Chain-of-Thought Supervised Fine-Tuning

Chain-of-Thought Supervised Fine-Tuning (CoT SFT) is a post-training paradigm in which a language model is optimized on targets that explicitly contain intermediate reasoning steps in addition to final answers. In the survey formulation, a chain of thought is represented as a sequence of reasoning nodes, and CoT fine-tuning is written as learning from triples of input, reasoning chain, and answer, with the model optimized on the concatenated reasoning-plus-answer target [2510.13170]. Across recent work, CoT SFT functions both as a direct method for transferring reasoning ability from stronger teachers to smaller or non-reasoning models and as a cold-start stage for later reinforcement learning, reflection, or long-context grounding [2505.03469] [2605.20201] [2601.21909]. The same literature also shows that CoT SFT is not a single technique but a family of supervision designs that differ in rationale format, target granularity, token weighting, representation editing, and the relation between visible reasoning and final answers [2510.27462] [2512.21017] [2507.10085].

## 1. Definition, scope, and canonical objective

In the survey account, CoT fine-tuning aims to “endow large language models with reasoning capabilities by training them on curated reasoning traces” [2510.13170]. A single CoT is formalized as
\[
\begin{aligned}
& \mathcal{C} = \{n_k = \{s_k(r_{k-1}), r_k\} | k=0,1,...K\} \\
& \quad\quad\quad\quad s.t. \quad \forall i < K, n_i \rightarrow n_{i+1}; \\
& \quad\quad\hspace{0.3cm} s_k(r_{k-1}) = r_k
\end{aligned}
\]
and CoT fine-tuning overall is written as
\[
\theta^* = \arg\min_{\theta} \mathbb{E}_{(x, y, c) \sim \mathcal{D}} \left[ \mathcal{L}(f_\theta(x), c \oplus y) \right]
\]
where \(x\) is the question, \(c\) the reasoning chain, and \(y\) the answer [2510.13170].

A closely related token-level formulation appears in work on answer-focused reweighting. Standard supervised fine-tuning is written as
\[
\mathcal{L}_{\text{SFT}}(\bm{\theta}) = -\sum_{i=1}^{N} \sum_{t=1}^{L_i} \log P\left(y_{i,t} \mid \bm{x}_i, \bm{y}_{i,<t}; \bm{\theta}\right)
\]
and in CoT-style settings the target sequence typically contains a reasoning segment followed by a final-answer segment [2512.21017]. In multimodal CoT-SFT, the same pattern appears explicitly as rationale tokens \(r^\star\) followed by answer tokens \(y^\star\), optimized under autoregressive negative log-likelihood conditioned on image and question [2606.01558].

Within this literature, CoT SFT is distinct from answer-only supervised fine-tuning, distinct from inference-time CoT prompting, and distinct from reinforced fine-tuning. The survey identifies pre-thinking, post-thinking, and multi-task learning as the main supervised regimes [2510.13170]. Other papers sharpen the distinction empirically: inference-time CoT prompting may improve alignment or reasoning only slightly, whereas supervised fine-tuning on explanations or reasoning traces can materially shift model behavior [2503.02976]. This suggests that CoT SFT is best understood as training on reasoning trajectories, not merely eliciting them.

## 2. Supervision formats and target structures

The dominant supervised format is pre-thinking: the model generates reasoning first and the answer afterward. The survey writes this as optimization on \(c \oplus y\), whereas post-thinking instead uses \(y \oplus c\), and multi-task learning separates rationale generation and label prediction through different prefixes [2510.13170]. These distinctions matter because they affect whether reasoning serves as a scaffold for answer generation or as a post hoc explanation.

Recent papers use several structured target formats. In SFTKey, responses are explicitly segmented as
\[
\hat{\bm{y}}_i = [\texttt{<Thinking>} \; \bm{y}^{(\text{think})}_i \; \texttt{</Thinking>} \; \texttt{<Answer>} \; \bm{y}^{(\text{answer})}_i \; \texttt{</Answer>} ]
\]
and the second training stage masks out the thinking span so that only answer tokens contribute to the loss [2512.21017]. In SoftCoT, the training sequence is
\[
[\mathbf{p}_{\text{task}},\mathcal{Q},\mathbf{t}_{\text{inst}},\mathcal{R},\mathcal{A}]
\]
with loss applied over the rationale span \(\mathcal{R}\) and answer span \(\mathcal{A}\), while pre-rationale tokens are masked [2502.12134]. In long-context finance reasoning, the supervised target explicitly contains “properties, sub-answers, and final answers,” and the objective counts tokens in both reasoning and answer spans [2502.13127].

The literature also varies the granularity of what counts as a reasoning trace. Some methods use full natural-language derivations with calculations, as in conventional math CoT distillation [2505.03469]. Some deliberately abstract away calculations: Solution Guidance Fine-Tuning trains on “problem-solving guidance without calculations or extra explanations,” and CoMT trains on “abstract meta-thoughts excluding numerical calculations” [2412.09906] [2601.21909]. Other methods impose schema-like reasoning stages. Fine-R1 structures targets as “visual analysis, candidate subcategories, comparison, and final prediction,” each enclosed in dedicated tags [2602.07605]. ProxyCoT uses proxy-derived traces as full-sequence SFT targets on long contexts, without separate symbolic grounding or step-specific labels [2605.20201].

A recurring issue is whether all target tokens should be treated equally. Standard CoT SFT uses uniform token-level cross-entropy. Several later methods argue that this is suboptimal because long rationales dominate the loss. SFTKey addresses this by a second answer-only stage [2512.21017]. VCORE instead derives token weights from first-order gradient utility and constrains the weighting distribution to remain close to uniform for stability [2510.27462]. This suggests that CoT SFT has shifted from a purely sequence-level perspective toward more selective supervision over which reasoning tokens matter most.

## 3. Data construction and sources of reasoning traces

CoT SFT depends on explicit reasoning traces, and the literature uses both manual and automatic construction. The survey divides data collection into manual annotation and automatic annotation, with automatic pipelines further split into LLM-based and rule-based generation [2510.13170]. Manual explicit traces appear in datasets such as GSM8K and MATH; implicit rationale-style supervision appears in evidence datasets such as e-SNLI and FEVER [2510.13170].

Teacher-generated CoT is central to modern practice. LS-Mixture SFT begins from “Chain-of-Thought (CoT) reasoning data distilled from large reasoning models (e.g., DeepSeek R1)” and mixes long CoTs with short counterparts obtained by structure-preserved rewriting [2505.03469]. Fine-R1 uses Qwen2.5-VL-32B to generate rationales, then samples repeatedly until the CoT leads to an exactly matched subcategory and manually verifies prediction consistency [2602.07605]. SoftCoT relies on tasks “that include annotated reasoning steps,” but the learned conditioning is continuous rather than discrete [2502.12134]. ARES uses teacher-provided sentence-level scores for RL and teacher-corrected rationales as SFT targets, so its supervised stage is built from repaired model outputs rather than raw gold traces [2407.00087].

Long-context reasoning papers emphasize a second problem: obtaining high-quality CoT on the full input is expensive or unreliable. ProxyCoT addresses this by collecting traces on a compact proxy context \(C^p\), either from a teacher or from RL on the proxy, then supervising the same trace on the full context \(C\) [2605.20201]. LongFinanceQA uses a multi-stage annotator, Property-driven Agentic Inference, to generate structured reasoning consisting of property extraction, retrieval-based sub-findings, and final synthesis [2502.13127].

The provenance and quality of traces are not uniform across papers. Some datasets use LM-generated CoT steps, as in Math10K and Commonsense60K for CRFT [2507.10085]. Some papers explicitly note underspecified provenance: the NL2SQL CoT study states that it “created a specialized dataset including detailed Chain-of-Thought (CoT) steps” but does not clearly specify whether those annotations were manual, generated by a larger model, or programmatically derived [2603.22942]. A plausible implication is that data curation, verification, and trace quality remain first-order variables in CoT SFT, not merely preprocessing details.

## 4. Major methodological variants

A large part of recent work modifies standard CoT SFT rather than discarding it. One line addresses verbosity and overthinking. LS-Mixture SFT combines long CoT reasoning data with short rewritten counterparts and reports that, relative to direct SFT, it improves average accuracy by 2.3% while reducing response length by approximately 47.61% [2505.03469]. SFTKey retains full CoT training in Stage 1 but then fine-tunes only the final-answer span, with the stated goal of balancing long rationale learning against the “Key” portion that determines benchmark correctness [2512.21017].

A second line changes the representation on which supervision operates. SoftCoT keeps the backbone LLM frozen and learns a projection from assistant-generated soft thought tokens into the target model’s representation space, while still applying next-token loss over rationale and answer spans [2502.12134]. CRFT treats CoT SFT as a representation-editing problem: it identifies hidden states whose perturbation changes correctness and optimizes only those critical representations through a low-rank intervention,
\[
\Phi(\boldsymbol h) = \begin{cases} \boldsymbol h+\boldsymbol R^T(\boldsymbol W\boldsymbol h + \boldsymbol b - \boldsymbol R\boldsymbol h), & \text{if }\boldsymbol h \in \boldsymbol M(\boldsymbol h)\\
\boldsymbol h, & \text{otherwise.} \end{cases}
\]
with the base model frozen [2507.10085].

A third line modifies the supervision target itself. CoMT argues that full worked trajectories entangle abstract strategy with problem-specific execution, so it trains on “meta-thought” trajectories produced by a teacher under the prompt “Do NOT calculate any specific numbers” [2601.21909]. SGFT similarly replaces full CoT with high-level “Solution Guidance,” which encodes problem decomposition without calculations or detailed derivations [2412.09906]. These methods do not reject supervised reasoning traces; they alter the level of abstraction at which supervision is supplied.

A fourth line tackles token allocation. VCORE defines token utility as
\[
s_t(x,y,\theta) = \big\langle \nabla \mathcal L(\theta), \nabla_\theta \ell_t(\theta;x,y)\big\rangle
\]
and chooses token weights by solving
\[
\max_{q\in \Delta}\ \sum_{t\ge 1} q(t)\, s_t(x,y) \quad \text{s.t.}\quad \mathrm{KL}(q\|u)\le \delta
\]
which yields a Gibbs weighting distribution over token positions [2510.27462]. This reframes CoT SFT as adaptive supervision allocation rather than uniform token imitation.

A fifth line integrates CoT SFT with subsequent policy optimization. ThinkDrive treats CoT SFT as the cold-start stage for autonomous-driving RL [2601.04714]. Fine-R1 uses CoT SFT to build a strong open-world fine-grained visual classifier before Triplet Augmented Policy Optimization [2602.07605]. ProxyCoT uses SFT itself as the second-stage transfer mechanism after proxy-context trace acquisition [2605.20201]. ARES alternates RL and SFT, where SFT trains on teacher-corrected rationales after RL-induced exploration [2407.00087]. These papers collectively position CoT SFT as both a standalone method and an initialization layer for broader post-training pipelines.

## 5. Empirical effects across domains

The empirical record is mixed but substantial. In teacher-distilled settings, CoT SFT can transfer reasoning capability effectively. LS-Mixture SFT reports that, compared to direct SFT, models achieve an average accuracy improvement of 2.3% across benchmarks with much shorter outputs [2505.03469]. In NL2SQL, adding explicit CoT reasoning to Qwen fine-tuning improves execution accuracy from 45.33% under standard SFT to 54.50% under CoT fine-tuning on the reported 600-query benchmark [2603.22942]. In long-context finance reasoning, CoT-supervised LongPAI improves overall Average Score on the Loong financial subset from 45.88 to 70.54, while the answer-only ablation LongPAI\(^\S\) degrades sharply at longer context ranges [2502.13127].

Small-model studies often present CoT SFT as especially valuable or, conversely, as something that requires simplification. SGFT reports that high-level guidance can outperform CoT-fine-tuned baselines for 6B–7B models under limited-data regimes [2412.09906]. CRFT shows that CoT-bearing supervision combined with targeted representation editing can raise GSM8K accuracy on LLaMA-2-7B from 14.6 to 32.8 while using only 0.016% trainable parameters [2507.10085]. SoftCoT reports gains on five reasoning benchmarks while freezing both assistant and backbone and training only a projection module [2502.12134].

Long-context and multimodal settings show both the promise and the limits of standard CoT SFT. ProxyCoT demonstrates that SFT on proxy-derived traces conditioned on the full long context can substantially outperform direct long-context baselines on SciTrek and HotpotQA [2605.20201]. Fine-R1 reports that CoT SFT alone improves relative semantic similarity by 13.30% in its open-world FGVR ablation, and that direct CoT prompting without training has only limited impact [2602.07605]. By contrast, a multimodal analysis across Qwen2.5-VL, InternVL3.5, and Gemma3 families shows that off-the-shelf CoT prompting often underperforms direct prompting, and that standard CoT-SFT only partially fixes premature answer commitment and weak visual-token access [2606.01558].

The literature also documents settings where standard fine-tuning or CoT prompting underperform. In exception handling, CoT prompting reduces the human–model refusal-rate gap only slightly, from 45.3% to 40.6%, whereas explanation-based SFT reduces it to 15.9% on the reported scenario set [2503.02976]. In long CoT teacher comparison, lower SFT loss on DeepSeek-R1-0528 traces yields worse generalization than higher-loss training on gpt-oss-120b traces, indicating that “correct” trajectories can still encode inefficient reasoning patterns [2604.01702]. And a diagnostic study of standard downstream fine-tuning finds that CoT answer accuracy and CoT faithfulness often decrease after task adaptation, particularly for smaller models [2411.15382]. This suggests that gains in end-task accuracy do not automatically imply gains in reasoning quality.

## 6. Limitations, controversies, and open directions

Several limitations recur across the literature. The first is overthinking and verbosity. Teacher-distilled long-CoT models often inherit redundant exploratory behavior, verbose chains, or excessive token costs [2505.03469] [2604.01702]. The second is supervision imbalance: long rationale spans dominate the token-level objective even though the final answer is short and benchmark-critical [2512.21017] [2510.27462]. The third is faithfulness. Generated CoTs may be post hoc, weakly causal, or only partially grounded, especially in multimodal systems or after unrelated task fine-tuning [2606.01558] [2411.15382].

A fourth issue is data quality. The survey repeatedly stresses that synthetic traces require filtering, verification, or external checking [2510.13170]. Fine-R1 uses repeated sampling and manual verification [2602.07605]. ProxyCoT keeps only proxy traces that produce correct answers [2605.20201]. ARES builds SFT targets from teacher-corrected rationales precisely because RL outputs are unstable [2407.00087]. The long-CoT teacher study sharpens this further: answer correctness alone is insufficient if the reasoning pattern is branch-heavy and redundant [2604.01702].

A fifth issue concerns generalization and what exactly should be supervised. CoMT argues that complete trajectories are the wrong grain because they entangle reusable strategy with instance-specific execution [2601.21909]. SGFT makes a related claim for small models and low-data settings [2412.09906]. SFTKey, VCORE, and CRFT all imply that standard full-sequence cross-entropy is too coarse, but they intervene at different levels: answer spans, token weights, or critical representations [2512.21017] [2510.27462] [2507.10085]. This suggests that “CoT SFT” is evolving from uniform imitation of full traces toward more structured, selective, and task-aware supervision.

Open directions are correspondingly broad. The survey calls for more faithful, diverse, reflective, transferable, efficiently internalized, and fact-grounded traces [2510.13170]. Multimodal work suggests supervising reasoning dynamics, not just rationale text [2606.01558]. Long-context work suggests transferring traces from easier proxy contexts or agentic decompositions [2605.20201] [2502.13127]. Representation- and utility-based methods suggest that future CoT SFT may operate less through blanket weight updates and more through targeted editing or optimization-theoretic token allocation [2507.10085] [2510.27462]. A plausible implication is that CoT SFT is no longer defined solely by visible step-by-step text, but by a broader design question: which parts of a reasoning process should be supervised, at what granularity, and with what relation to final-answer correctness.

Source: https://www.emergentmind.com/topics/chain-of-thought-supervised-fine-tuning