---
title: CoT-Collection Dataset Overview
url: https://www.emergentmind.com/topics/cot-collection-dataset
type: topic
---

# CoT-Collection Dataset Overview

The CoT-Collection Dataset is a large-scale, instruction-tuning resource designed to equip language models with explicit chain-of-thought (CoT) reasoning capabilities across a broad spectrum of tasks. It was introduced to address the inherent limitations of smaller (<100B parameter) language models in generating multi-step rationales, enabling these models to approach the zero-shot and few-shot reasoning capacities heretofore exclusive to much larger models. The dataset serves both as a pretraining corpus for intermediate rational supervision and as a benchmark for research on reasoning transfer, task generalization, and template learning behaviors in natural language processing.

## 1. Dataset Composition and Structure

The CoT-Collection comprises 1,060 distinct instruction-formatted tasks, covering 1.84 million examples, each annotated with 1–5 CoT rationales. Tasks are distributed among 26 “families,” including multiple-choice QA, extractive QA, arithmetic word problems, commonsense reasoning, natural-language inference, symbolic logic, list-manipulation, dialogue, and code-oriented domains. Unlike the original Flan Collection, which included only nine CoT-augmented tasks, the CoT-Collection offers comprehensive CoT coverage across domains.

Table: Task Family Distribution (Selected)

| Family                 | #Tasks | #Examples |
|------------------------|--------|-----------|
| Multi-choice QA        | 220    | 420K      |
| Extractive QA          | 180    | 300K      |
| Arithmetic Reasoning   | 105    | 200K      |
| Commonsense (SNI)      | 150    | 250K      |
| NLI (FLAN)             | 80     | 150K      |
| Logic & Symbolic       | 75     | 120K      |
| Dialogue/Code/Others   | 250    | 400K      |

Each dataset example is a JSONL record with fields: `{task_id, instruction, input, chain_of_thought, answer}` stored in ten ≈184K-instance shards. Examples are constructed using prompt instruction, input (possibly empty), CoT rationale (with explicit answer token), and ground-truth answer.

## 2. Annotation and Generation Pipeline

Annotation follows a unified, partly human-curated and largely model-based process:

1. **Task family grouping**: Each task is associated with a demonstration set D^T_k (typically 6–8 high-quality, hand-written rationales by authors).
2. **Rationale generation**: OpenAI Codex (code-davinci-002) is prompted with the "Let's think step by step" phrase in few-shot in-context style, conditioning on ground-truth label.
3. **Decoding**: Nucleus (top-p=0.8) sampling and no-repeat ngram enforcement, generating five rationale candidates per example.
4. **Filtering**: Post-processing retains rationales containing the gold answer, with length <512 tokens, unique content, and absence of code degeneration or repeated sentences.

Manual A/B testing and filtering are performed to ensure fluency and informativeness of initial demonstration rationales; all further rationale generation is model-driven according to the described criteria.

## 3. Usage and Model Fine-Tuning Protocol

For supervised fine-tuning, representative code examples are provided:

```python
from datasets import load_dataset
ds = load_dataset('json', data_files='cot_collection/*.jsonl', split='train')
def preprocess(ex):
    prompt = ex['instruction'] + '\n' + ex['input'] + '\nLet\'s think step by step.'
    return {'prompt': prompt, 'labels': ex['chain_of_thought'] + ' [ANSWER] ' + ex['answer']}
train_ds = ds.map(preprocess, remove_columns=ds.column_names)
```

Fine-tuning is executed with hyperparameters—Flan-T5-3B: AdamW, batch 64, lr=5e-5, accumulation 8, 1 epoch; Flan-T5-11B: Adafactor, batch 8, lr=1e-4. Mini-batches should be sampled across sources to ensure distributional uniformity (e.g., FLAN 23.9%, P3 30.9%, SNI 25.5%, etc.). Efficient few-shot adaptation is possible via LoRA (rank=4).

## 4. Evaluation, Metrics, and Results

CoT-Collection enables both zero-shot and few-shot evaluations. The primary metric is accuracy:

\[
\text{Accuracy} = \frac{\#\text{Correct}}{\#\text{Total}} \times 100\%
\]
\[
\Delta \text{Accuracy} = \text{Accuracy}_{\text{CoT-tuned}} - \text{Accuracy}_{\text{base}}
\]

On BIG-Bench-Hard (27 tasks):

| Model                   | Direct | CoT Eval | ΔCoT vs Flan |
|-------------------------|--------|----------|-------------|
| Flan-T5-3B              | 37.1%  | 34.1%    | —           |
| CoT-T5-3B (Ours)        | 36.2%  | 38.4%    | +4.34%      |
| Flan-T5-11B             | 41.0%  | 38.6%    | —           |
| CoT-T5-11B (Ours)       | 42.6%  | 42.2%    | +2.60%      |

Few-shot (64-shot, 4 domains; using LoRA):

| Model                      | #Params                | Avg Acc | Δ vs. Flan |
|----------------------------|------------------------|---------|------------|
| Flan-T5-3B full FT         | 2.8B                   | 61.78%  | —          |
| CoT-T5-3B + LoRA CoT-FT    | 2.35M                  | 64.02%  | +2.24%     |
| Flan-T5-11B + LoRA FT      | 4.72M                  | 66.59%  | —          |
| CoT-T5-11B + LoRA CoT-FT   | 4.72M                  | 68.96%  | +2.37%     |
| ChatGPT+ICL (64 demos)     | —                      | 54.98%  |            |

Models fine-tuned on the CoT-Collection attain better zero-shot generalization and show marked improvements over equivalent-size, standard-flan-trained and ICL-only baselines.

## 5. Scientific Insights and Best Practices

Key findings:

- Explicit CoT supervision is necessary for teaching how to decompose and solve complex, multi-step problems; pure in-context learning with CoT prompts is insufficient for smaller models.
- Diversity of task types in training, more so than volume, is essential for robust reasoning transfer; 10,000 diverse CoT examples from 1,060 tasks yield better generalization than 180,000 examples over just nine tasks.
- Positive transfer is observed across task families, with no visible catastrophic forgetting on reused tasks.
- LoRA adaptation enables efficient few-shot CoT learning with minimal parameter overhead.
- Filtering rationales for answer-presence, brevity, and non-degeneration is a practical necessity; code-based filters are included in the release.

## 6. Limitations and Directions for Extension

CoT-Collection is exclusively English; current multilingual zero-shot performance is near zero for Korean, Chinese, and Japanese. Rationale generation depends on the (proprietary) Codex model—future iterations may leverage open-source LLMs or more advanced multi-path rationales such as "Tree of Thoughts". It is recommended to extend to non-English settings, and to experiment with other approaches to CoT rationalization and evaluation protocols.

Reliance on human-crafted demonstrations for each family is a critical design choice—expanding this to cover more instruction types or to support semi-automatic CoT generation in new domains would be a natural extension.

## 7. Availability and Impact

The dataset, code, and model checkpoints are publicly accessible. CoT-Collection is, to date, the largest and most diverse open instruction-tuning resource with CoT supervision for language model reasoning. It serves as a reference corpus for model pretraining, as a basis for research in reasoning and transfer, and as a framework for future advances in medium- and low-parameter LM chain-of-thought generalization [2305.14045].

Source: https://www.emergentmind.com/topics/cot-collection-dataset