SwimBird-SFT-92K Multimodal CoT Dataset

• SwimBird-SFT-92K is a curated dataset of 92,300 multimodal chain-of-thought examples that supports dynamic switching among text-only, vision-only, and interleaved reasoning modes.
• The dataset is constructed using a rigorous three-stage pipeline that filters and assigns reasoning modes based on pass@8 accuracy and intermediate hint evaluations.
• Models fine-tuned on SwimBird-SFT-92K have achieved state-of-the-art improvements across multiple benchmarks, demonstrating enhanced robustness in both visual and textual tasks.

SwimBird-SFT-92K is a large-scale, curated dataset comprising 92,300 chain-of-thought (CoT) multimodal reasoning examples, explicitly designed to enable dynamic, input-adaptive switching among three distinct reasoning patterns—text-only, vision-only, and interleaved vision-text—within Multimodal LLMs (MLLMs). Developed to support and evaluate the SwimBird reasoning-switchable MLLM architecture, this dataset systematically integrates textual and visual reasoning trajectories and enables fine-grained supervision of query-conditioned reasoning modality (Tong et al., 5 Feb 2026).

1. Dataset Composition

SwimBird-SFT-92K (|SwimBird-SFT-92K| = 92,300) consists of samples stratified by reasoning mode as follows:

• Text-only (T): $N_T = 50{,}000$ (54.2%)
• Vision-only (V): $N_V = 8{,}800$ (9.5%)
• Interleaved (I): $N_I = 33{,}500$ (36.3%)

The dataset sources and their mode-wise contributions are shown in the table below:

Source	Total	Text-only	Vision-only	Interleaved
Zebra-CoT	26,300	0	5,900	20,400
ThinkMorph	7,100	0	1,200	5,900
MathCanvas-Instruct	8,900	0	1,700	7,200
OpenMMReasoner	50,000	50,000	0	0

Text-only data stems from OpenMMReasoner and targets general VQA and math question answering, while vision-only and interleaved modalities predominantly focus on tasks such as visual search, jigsaw puzzles, mazes, geometry, chart reasoning, and spatial navigation, drawing from Zebra-CoT, ThinkMorph, and MathCanvas-Instruct.

2. Construction and Reasoning-Mode Curation Pipeline

The curation pipeline for SwimBird-SFT-92K proceeds through three delineated stages:

Stage 1: Candidate Collection and Filtering

All candidate multimodal CoT samples ($S_0$) are aggregated from Zebra-CoT, ThinkMorph, and MathCanvas. For each example $x$, pass@8 accuracy is evaluated via Qwen3VL-8B on the original question-image pair ($\mathrm{pass}_\mathrm{base}(x)$). Trivial ("easy") samples—where $\mathrm{pass}_\mathrm{base}(x) = 1.0$—are filtered out, yielding $S_1$.

Stage 2: Reasoning-Mode Assignment

For $x \in S_1$, pass@8 is computed in the presence of intermediate "thinking" images ($\mathrm{pass}_\mathrm{hint}(x)$). Only examples with $N_V = 8{,}800$0 are retained. Modes are then assigned by: $N_V = 8{,}800$1 This produces $N_V = 8{,}800$2 high-quality multimodal samples.

Stage 3: Integration of Text-Only CoT

$N_V = 8{,}800$3 text-only CoT sequences from OpenMMReasoner, already filtered by pass@8, are incorporated. The final dataset thus comprises 92,300 documents, with empirical mode frequencies $N_V = 8{,}800$4 for $N_V = 8{,}800$5.

No explicit reweighting is applied—mode proportions reflect source and filtering outcomes.

3. Annotation, Quality Control, and Verification

• Reasoning Paths: Textual chains are sourced directly from human- or model-curated CoT traces in contributing datasets. Visual "hints"—intermediate images portraying the reasoning process—are also inherited from source datasets (e.g., diagram crops, annotated sketches).
• Verification: An instance is accepted only if $N_V = 8{,}800$6, ensuring that intermediate hints do not degrade answerability. Label correctness is assessed automatically using Qwen3-235B-Instruct by comparing prediction against the gold answer.
• Quality Gating: The pass@8 metric is employed for exclusion of low-quality or irrelevant CoT samples. No manual re-annotation is required, as automated filters achieve high-precision instance selection (Tong et al., 5 Feb 2026).

4. Statistical and Structural Properties

• Chain Lengths: Mean textual chain length is $N_V = 8{,}800$7 tokens for text-only samples (σ ≈ 15) and $N_V = 8{,}800$8 tokens (σ ≈ 12) for interleaved examples.
• Visual Token Budget: Each instance enforces a variable latent-token span ($N_V = 8{,}800$9) with $N_I = 33{,}500$0 and $N_I = 33{,}500$1. Vision-only samples average $N_I = 33{,}500$2 (σ ≈ 8); interleaved samples $N_I = 33{,}500$3.
• Resolution Scaling: High-resolution inputs yield $N_I = 33{,}500$4 near $N_I = 33{,}500$5; low-resolution inputs near $N_I = 33{,}500$6.
• Mode Distribution Across Benchmarks: For DynaMath and MathVerse_MINI, text-only constitutes ~90% of instances; for V* Bench and HR-Bench (4K/8K), the split is approximately 40% vision-only, 30% interleaved, and 30% text-only, indicating adaptive mode selection by benchmark.

5. Integration in Supervised Fine-Tuning

• Base Model: Qwen3-VL-8B (encoder-decoder) with frozen vision encoder.
• SFT Objective: Hybrid autoregressive loss combining next-token prediction for text and next-embedding prediction for visual latents:

$N_I = 33{,}500$7

with $N_I = 33{,}500$8, $N_I = 33{,}500$9.

• Training Regime: Batch size 128, cosine learning rate schedule (initial LR $S_0$0), on A100–80G GPUs. During inference, the dynamic latent span is terminated by emission of the </latent> token.

6. Comparative Context and Performance Impact

SwimBird-SFT-92K is the first multimodal CoT dataset constructed to supervise three reasoning patterns (text-only, vision-only, and interleaved) in a balanced, query-adaptive fashion. Its scale and diversity surpass predecessors such as Zebra-CoT (26.3K), ThinkMorph (7.1K), and MathCanvas-Instruct (8.9K), as well as text-only OpenMMReasoner (50K). Prior datasets are limited to fixed reasoning patterns or static latent budgets, precluding dynamic, sample-adaptive reasoning.

Empirically, models fine-tuned on SwimBird-SFT-92K demonstrate state-of-the-art results across multiple benchmarks, with absolute improvements of +1.2 (V* Bench), +2.0 (HR-Bench 4K), +3.6 (HR-Bench 8K), +6.5 (MMStar), +10.7 (WeMath), and +1.9 (DynaMath) over prior best methods. This suggests that explicit tri-mode supervision and dynamic latent allocation substantially enhance robustness and efficacy on both textual and vision-intensive tasks (Tong et al., 5 Feb 2026).

In summary, SwimBird-SFT-92K constitutes a principled, high-quality, and large-scale dataset supporting the development of MLLMs capable of flexible, context-driven multimodal reasoning.