---
title: Synthetic CoT-Supervised Data
url: https://www.emergentmind.com/topics/synthetic-cot-supervised-dataset
type: topic
---
# Synthetic CoT-Supervised Data
Synthetic CoT-supervised datasets are training resources in which the supervision target includes intermediate reasoning in addition to a final answer, while the reasoning itself is commonly programmatically generated, model-generated, distilled from existing corpora, transformed from structured annotations, or filtered from automatically produced traces rather than written entirely from scratch by human annotators. Recent work spans mixed curated and model-generated video reasoning, synthetic long-context finance reasoning, large distilled reasoning corpora, label-free biological trace filtering, semi-automatic medical workflow traces, and interleaved vision-language reasoning, indicating that “synthetic” in this literature usually refers to the provenance of the reasoning supervision rather than to wholly fabricated problems or wholly human-authored rationales [2506.08817; 2502.13127; 2505.10937; 2507.16746].
## 1. Conceptual scope and taxonomy
The category is not uniform. Some works introduce standalone datasets whose primary contribution is a new reasoning corpus, while others contribute a supervision construction method or a training framework that depends on synthetic CoT data. LongFinanceQA is explicitly introduced as a synthetic dataset in finance whose labels contain intermediate reasoning before the final conclusion [2502.13127]. OmniThought is presented as a large synthetic supervision resource built from existing reasoning problems plus teacher-generated and revalidated CoTs, with additional metadata describing reasoning verbosity and cognitive difficulty [2505.10937]. Zebra-CoT is a mixed multimodal corpus containing interleaved text-image traces, with both curated real-world traces and synthetic examples generated from templates filled in by VLMs [2507.16746].
Other resources occupy a mixed or semi-automatic position rather than a purely synthetic one. Video-CoT is best characterized as a mixed, partially synthetic Chain-of-Thought-supervised video dataset: its 192,000 spatiotemporal QA pairs are largely curated or derived from public datasets and processed annotations, while its 23,000 CoT samples are generated by a VLM and then filtered and manually reviewed [2506.08817]. Step-CoT is similarly not a fully synthetic clinical corpus; its cases and reports are real, its seven-step reasoning chains are generated from those reports using DeepSeek-R1 under a fixed schema, and all diagnostic procedures and data labels are reviewed and verified by a board-certified physician [2603.13878].
A separate class of papers does not primarily release a new dataset, but instead repurposes or transforms synthetic CoT supervision. “From Explicit CoT to Implicit CoT” is explicitly a training-method paper rather than a new standalone benchmark dataset; its central contribution is a staged transformation from explicit CoT targets to progressively reduced supervision, chiefly on synthetic multiplication and augmented GSM8K [2405.14838]. SIM-CoT likewise does not introduce a new dataset, but depends centrally on GSM8K-Aug and converts explicit step sequences into latent-step supervision for implicit CoT models [2509.20317]. CoT-Self-Instruct is best understood as a synthetic prompt-generation and curation method that uses CoT during data creation; its resulting datasets are primarily prompt-answer or preference data rather than explicit rationale-supervision corpora in the usual sense [2507.23751].
This variation has produced a recurring ambiguity in the term “CoT-annotated.” In Video-CoT, for example, “CoT-annotated” does not mean fully human-written reasoning; it denotes Qwen2.5-VL-72B-Instruct-generated traces wrapped in `` and `` tags and then filtered and reviewed [2506.08817]. A plausible implication is that any taxonomy of synthetic CoT datasets must distinguish at least between fully programmatic traces, teacher-generated traces over existing labels, semi-automatic expert-curated traces, and methods that use CoT to create training prompts without supervising the final model on exposed rationales.
## 2. Provenance of reasoning supervision
Recent work uses several recurring sources for synthetic reasoning traces.
| Construction mode | Representative resources | Trace provenance |
|---|---|---|
| Programmatic derivation | [2405.14838], [2502.15132] | Long multiplication steps; DAG-plus-MLP token chains |
| Teacher- or VLM-generated traces | [2506.08817], [2505.10937], [2507.16746] | Qwen2.5-VL-72B-Instruct; DeepSeek-R1 and QwQ-32B; Gemini-2.5 and GPT-4.1 |
| Agentic generation over source documents | [2502.13127], [2507.12901] | PAI property extraction, retrieval, summarization; MKE plus SCR |
| Semi-automatic expert-curated traces | [2603.13878], [2605.30062] | Report-derived step chains with physician review; Gemini drafts refined by human experts |
| Label-free selection of synthetic traces | [2510.05871] | Gemini 2.5 Pro traces retained by low uncertainty |
The most direct form of synthetic CoT supervision is programmatic derivation. In the multiplication setting of “From Explicit CoT to Implicit CoT,” examples are generated algorithmically from long multiplication, with reversed inputs, reversed intermediate partial products, and reversed outputs; for \(12 \times 34\), the paper gives input `2 1 * 4 3`, CoT `8 4 + 0 6 3`, and output `8 0 4` [2405.14838]. CoT-ICL Lab generalizes this idea to a synthetic tokenized setting in which each chain token \(y_c\) is produced by a causal selector \(g_c\) and a token processor \(h_c\), yielding programmable rationale length, dependency width, and function diversity [2502.15132].
Teacher-generated traces dominate many large corpora. Video-CoT constructs a pipeline for CoT data generation using Qwen2.5-VL-72B-Instruct over video inputs, prompts, and examples, producing structured reasoning outputs in `` tags [2506.08817]. OmniThought uses DeepSeek-R1 and QwQ-32B as teacher large reasoning models to produce multiple CoTs per problem and then revalidates them with an LLM-as-a-judge pipeline [2505.10937]. Zebra-CoT uses Gemini-2.5 and GPT-4.1 to fill placeholders in templated multimodal traces or to enhance raw reasoning traces into logically coherent interleaved text-image narratives [2507.16746].
Agentic generation pipelines move beyond one-pass prompting. LongFinanceQA uses Property-driven Agentic Inference, a three-stage framework of property extraction, property-based retrieval, and summarization, to create reasoning-augmented answers from annual reports [2502.13127]. Agentar-DeepFinance-300K first expands the QA space through Multi-perspective Knowledge Extraction with Q2A, A2Q, and T2Q branches, then applies Self-Corrective Rewriting so that failed initial CoT samples can be diagnostically reflected on, rewritten, and reverified against golden answers instead of discarded [2507.12901]. D-SCoRE applies a three-stage document-centric generation pipeline from arbitrary text: QA generation, QA quality control with KEEP/DELETE/TYPEFIX decisions, and counterfactual material generation with three distractors per QA pair [2508.01309].
Semi-automatic expert-curated pipelines typically start from natural source data and synthesize only the reasoning layer. Step-CoT converts real chest X-ray reports into a seven-step diagnostic chain aligned to clinical workflow and then validates the outputs clinically [2603.13878]. FakeClue++ combines real/fake labels, artifact localization, explanatory checklists, and physical-law-oriented descriptions, then uses LLMs to rephrase those interpretable annotations into diverse SFT formats and multi-turn dialogue variants [2605.30062].
Label-free filtering forms a distinct provenance regime. In biological perturbation prediction, one greedy trace and eight sampled traces are generated for each unlabeled example, uncertainty is estimated from semantic consistency plus predictive perplexity through CoCoA, and only low-uncertainty traces are retained, with per-class filtering to preserve coverage across outcomes [2510.05871]. This suggests a synthetic CoT corpus need not depend on ground-truth labels if confidence signals are used as a proxy for supervision quality.
## 3. Construction pipelines and dataset engineering
Synthetic CoT datasets are usually assembled as layered pipelines rather than as a single annotation step. Video-CoT illustrates this clearly. Its base supervision consists of 192,000 fine-grained spatiotemporal QA pairs organized into six subtasks—TVL, VC, SVG, STVG, SRR, and TVR—built from publicly available datasets such as VTimeLLM-stage2, MSR-VTT, MSVD, WebVid, HCSTVG-V1, and VidSTG. For grounding and reference tasks, the construction procedure selects query targets, ensures consecutive presence across frames, discards objects that appear for less than 2 seconds, derives start and end timestamps and pixel locations, maintains spatial relationships, and then creates spatiotemporal relational QA data. Only the 23,000 CoT samples form the reasoning-augmented subset, generated by a VLM and retained after task-specific thresholding plus manual expert review [2506.08817].
LongFinanceQA uses a different engineering strategy because its source material is long-form and bilingual. It is built from 6,911 annual reports in English and Chinese, collected from SEC-10-K and cninfo, restricted to reports published before 2022 and filtered to a token range of 20K to 80K. Its generation pipeline samples financial metrics and report metadata, creates single-source or multi-source questions, and then uses PAI to produce properties, retrieval-based sub-findings, and final summarized conclusions. The resulting dataset contains 46,457 QA pairs with CoT reasoning, with 45.6% single-source and 54.4% multi-source composition [2502.13127].
OmniThought scales the same general pattern to a broad reasoning corpus. It aggregates problems from OpenThoughts2-1M and DeepMath-103K, generates multiple CoTs with DeepSeek-R1 and QwQ-32B, validates reasoning and solution separately, and computes Reasoning Verbosity and Cognitive Difficulty scores. The release contains 2.059 million CoT processes for 708K reasoning problems, with at least two validated correct CoTs per problem. Unlike “perfect-only” corpora, it retains lower-quality CoTs unless they produce an incorrect answer, because answer-correct but reasoning-flawed traces can still support DPO or reward-model training [2505.10937].
Several later pipelines explicitly optimize the knowledge space before rationale generation. Agentar-DeepFinance-300K uses Q2A direct curation, A2Q counterfactual augmentation, and T2Q CoT knowledge mining, then samples multiple CoTs per QA pair and verifies answer consistency with a lightweight model; if the answer is wrong, SCR adds a reflection chain \(T_{rf}\), a rewritten chain \(T_{rw}\), and iterative re-verification [2507.12901]. D-SCoRE operates over arbitrary textual sources by segmenting documents, generating explicit and implicit questions, validating source fidelity and question type, and then generating four-option answer materials with one correct answer and three semantically close but factually wrong distractors [2508.01309]. Zebra-CoT combines sourced multimodal problems, synthetic task generation, rendered intermediate states, and VLM-written narratives, then filters invalid image-placeholder mappings to ensure that interleaved text-image traces can be automatically parsed [2507.16746].
These pipelines reveal a common engineering pattern. The underlying tasks are often not invented from nothing; instead, the corpus is expanded by curating source datasets, transforming their annotations, generating candidate rationales with strong teachers or programs, and then filtering or repairing those candidates. A plausible implication is that the decisive design variable is not “synthetic versus real” in isolation, but the exact layer at which synthetic reasoning is injected.
## 4. Supervision formats and objective functions
The defining formal property of a synthetic CoT-supervised dataset is that the target contains reasoning as well as a final answer. LongFinanceQA states this contrast directly by setting ordinary QA as \(p_{\theta}(\mathbf{A}\mid \mathbf{X}, \mathbf{Q})\) and reasoning-augmented supervision as \(p_{\theta}(\mathbf{R}, \mathbf{A}\mid \mathbf{X}, \mathbf{Q})\), with dataset construction factorized as \(p_{\theta}(\mathbf{R}\mid \mathbf{X}, \mathbf{Q}) \cdot p_{\theta}(\mathbf{A}\mid \mathbf{X}, \mathbf{Q}, \mathbf{R})\). Its rationale is not a free-form essay but a mixed-format sequence containing extracted properties, retrieval-stage sub-answers, and a final summarized conclusion [2502.13127].
Video-CoT makes the same distinction in multimodal form. It defines Answer-Supervised Fine-Tuning as \((V,Q)\rightarrow A\) and CoT-Supervised Fine-Tuning as \((V,Q)\rightarrow (R,A)\), where \(V \in \mathbb{R}^{T \times H \times W \times C}\) is the video, \(Q\) is the question, \(A\) is the answer, and \(R=\{r_1,r_2,\ldots,r_n\}\) is the reasoning chain. The paper describes curriculum learning from simple 2-step reasoning scenarios to sophisticated 5-step chains, and the CoT traces are structured reasoning outputs wrapped in `` tags [2506.08817].
Other works change the granularity rather than the basic rationale-plus-answer design. SIM-CoT aligns each latent token \(z_k\) with one explicit reasoning step \(s_k\) and trains an auxiliary decoder under
\[
p_\phi(s_{1:K}\mid z_{1:K})=\prod_{k=1}^{K} p_\phi(s_k\mid z_k),
\]
while the base LLM predicts the final answer \(a\) from the latent sequence. Its training source, GSM8K-Aug, removes natural-language chains and preserves structured mathematical expressions such as `<<12*3=36>><<9*2=18>><<17*2=34>><<36+18+34=88>>`, so the step targets are symbolic expression chains rather than free-form rationales [2509.20317].
“From Explicit CoT to Implicit CoT” treats rationale visibility as a curriculum variable. Starting from explicit CoT supervision
\[
\min_\theta -\log P_\theta(y, z_{1:m}\mid x),
\]
it progressively removes the first \(s(t)\) rationale tokens and trains on
\[
\min_\theta -\log P_\theta(y, z_{1+\min(s(t),m):m}\mid x),
\qquad
s(t)=\left\lfloor \Delta \frac{t}{T}\right\rfloor.
\]
The same examples therefore appear in three supervision regimes: full explicit CoT, partial CoT, and no visible CoT [2405.14838].
Multimodal work further diversifies the serialization format. Zebra-CoT uses a trace schema with `QUESTION`, `REASONING TRACE`, `THOUGHT n`, interleaved image placeholders such as `[reasoning_image_1]`, and a `FINAL ANSWER`, while Bagel-Zebra-CoT training additionally wraps textual reasoning in ` ... ` and answers in ` ... ` [2507.16746]. Step-CoT instead structures each case as JSON with `step`, `question`, `options`, `answer`, and `reasoning`, organized around a seven-step chest-radiograph workflow [2603.13878]. FakeVLM-R1 constrains the reasoning format during RL with `` and `` tags plus a dialectical pattern `[Clue] \rightarrow [Why real/fake] \rightarrow [If fake/real]` [2605.30062].
These differences matter because “CoT supervision” may denote natural-language rationale tokens, symbolic steps, structured JSON sub-decisions, interleaved image-text traces, or latent-step reconstruction targets. The commonality lies in exposing intermediate structure to the learner, not in any single surface format.
## 5. Modalities, domains, and benchmark roles
Synthetic CoT supervision now appears across a wide range of modalities and scales.
| Resource | Domain and modality | Reported scale |
|---|---|---|
| Video-CoT [2506.08817] | Video spatiotemporal understanding | 192,000 QA pairs; 23,000 CoT samples; 4,500 benchmark QA pairs |
| LongFinanceQA [2502.13127] | Long-context bilingual finance | 46,457 QA pairs with CoT over 6,911 reports |
| OmniThought [2505.10937] | Mathematics, coding, science, puzzles | 2.059M CoTs for 708K problems |
| Agentar-DeepFinance-300K [2507.12901] | Financial reasoning | 300K |
| Zebra-CoT [2507.16746] | Interleaved text-image reasoning | 182,384 samples |
| Step-CoT [2603.13878] | Medical VQA on chest X-ray | more than 10K cases and 70K VQA pairs |
The domain range extends beyond text-only reasoning. Video-CoT targets temporal boundaries, object trajectories, spatial relationships, and video-grounded descriptions [2506.08817]. Zebra-CoT spans scientific reasoning, 2D visual reasoning, 3D visual reasoning, and visual logic or strategic games, with tasks such as geometry, graph algorithms, jigsaw, embodied planning, chess, ciphers, and Connect Four [2507.16746]. Step-CoT aligns medical VQA to a seven-step radiological diagnostic workflow rather than to free-form rationalization [2603.13878]. FakeClue++ extends the idea to multimodal synthetic-image detection by combining artifact explanations with physical-law-based authenticity anchors for real images [2605.30062].
Benchmark roles also vary. Some datasets are primarily training corpora evaluated on external benchmarks. LongFinanceQA is used to fine-tune LongPAI and evaluated mainly on Loong and \(\infty\)Bench rather than on an internal benchmark split [2502.13127]. OmniThought likewise functions as a training pool whose effectiveness is tested on AIME2024, MATH500, GPQA-D, and LiveCodeBench V2 [2505.10937]. Other resources bundle a benchmark with the training corpus: Video-CoT provides 750 benchmark items per task across six tasks, totaling 4,500 video QA pairs [2506.08817], and FakeClue++ reserves a 4,000-image benchmark evenly distributed across real and fake samples in four categories [2605.30062].
Some datasets are deliberately filtered down before training. In biological perturbation prediction, the main synthetic pool contains 48k traces, but the main retained training set is only the top 10% per class, yielding 4.8k examples [2510.05871]. In Step-CoT, the filtered clinical corpus contains 10,068 frontal-view studies and is split 70%/15%/15% into 7,043 train, 1,507 validation, and 1,518 test cases [2603.13878]. This suggests that synthetic CoT corpora are often best understood as generated candidate pools plus a retention policy, rather than as a monolithic static dataset.
## 6. Empirical effects, limitations, and recurring controversies
The main empirical result across this literature is that rationale-bearing supervision often outperforms answer-only supervision, but the magnitude depends strongly on task structure. Video-CoT reports a baseline TVL temporal IoU of 4.4 for Qwen2.5-VL 3B, 8.3 after answer-only fine-tuning, and 19.7 after CoT-SFT; the same CoT-SFT run also reports VC MENTOR 15.4, SRR EM 16.9, TVR MENTOR 21.1, SVG sIoU 17.0, STVG tIoU 14.1, and STVG sIoU 9.2 [2506.08817]. LongPAI trained on LongFinanceQA with reasoning-augmented outputs markedly outperforms the answer-only LongPAI\(^{\S}\) on long contexts, with Set4 \(200K\)–\(250K\) performance of 60.36 versus 22.14 on the financial subset of Loong [2502.13127]. In multiplication, Stepwise Internalization raises GPT-2 Small from 0.00 on \(7\times7\) and \(9\times9\) under No CoT training to 0.95 and 0.99 under ICoT-SI, even though the deployed model no longer emits the intermediate steps [2405.14838]. SIM-CoT reports gains over prior implicit-CoT baselines, including +8.2 on GPT-2 over Coconut and +3.0 on LLaMA-3.1 8B over CODI [2509.20317]. In biology, supervised fine-tuning on the 10% uncertainty-filtered subset yields 0.57 accuracy, compared with 0.52 for all synthetic data and 0.48 for a random 10% subset [2510.05871]. Zebra-CoT improves Anole-7B from 4.2% to 16.9% on its held-out test set and reports up to +13% gains on standard VLM benchmarks [2507.16746].
At the same time, the literature repeatedly warns that answer correctness and rationale quality are not identical. OmniThought explicitly retains answer-correct traces even when reasoning is imperfect, because such traces may still be useful for DPO or reward-model training [2505.10937]. Agentar-DeepFinance-300K verifies answers against golden answers, but the paper itself notes that answer verification guarantees final correctness more than rationale faithfulness [2507.12901]. The label-free biology pipeline acknowledges a classic failure mode of self-judging systems: if the teacher is confidently wrong, low-uncertainty filtering can still retain a bad trace [2510.05871].
Documentation quality is another recurring controversy. Video-CoT does not provide exact CoT prompts, decoding settings, candidate counts, self-consistency details, inter-annotator agreement for review, or exact task-wise thresholds [2506.08817]. LongFinanceQA reports no human evaluation of extracted properties, sub-answers, or rationale faithfulness [2502.13127]. Step-CoT reports physician verification but არა systematic annotation-agreement statistics or a large-scale faithfulness audit [2603.13878]. Zebra-CoT emphasizes logical coherence and automatic filtering, yet does not report a dedicated human study of rationale quality or a quantitative decomposition of its synthetic and naturally sourced fractions [2507.16746]. A plausible implication is that many synthetic CoT corpora are stronger as supervision resources than as audited explanations.
Teacher bias is also structural. OmniThought relies on DeepSeek-R1 and QwQ-32B for generation, validation, and scoring [2505.10937]. SIM-CoT depends entirely on GSM8K-Aug, which is expanded partly through GPT-4 synthetic generation [2509.20317]. FakeClue++ combines Gemini-generated drafts, LLM rewrites, and a critic-driven RL stage, making its final reasoning behavior inseparable from the preferences and failure modes of those models [2605.30062]. This suggests that synthetic CoT datasets do not simply encode task knowledge; they also encode a teacher’s preferred decomposition style, verbosity, and implicit inductive biases.
For that reason, the most precise contemporary understanding of a synthetic CoT-supervised dataset is neither “a dataset with explanations” nor “a dataset fabricated by an LLM.” It is a training resource in which intermediate reasoning becomes an explicit supervision object through programmatic derivation, teacher generation, structured transformation, or filtered distillation, usually layered on top of pre-existing tasks or annotations. The field’s central question is therefore not whether the traces are synthetic, but how their provenance, filtering, granularity, and alignment to the target learner affect transfer, stability, interpretability, and error modes.
Source: https://www.emergentmind.com/topics/synthetic-cot-supervised-dataset