CoCo-10K: Dual-Domain Dataset Overview
- CoCo-10K is a dual-context dataset: one subset provides 10,000 spoken captions for speech research while the other offers structured Text–Code pairs for T2I tasks.
- The Speech-COCO subset features 10 hours of speech with detailed word-level annotations, controlled speed perturbation, and natural disfluencies to support unsupervised term discovery.
- The Code-as-CoT branch supplies explicit training for executable code generation and draft-to-final image refinement, leading to significant performance gains on T2I benchmarks.
In the literature supplied here, CoCo-10K has two explicit dataset-level meanings. In speech-and-vision research, it denotes the 10-hour / 10,000-spoken-caption portion of Speech-COCO used for a preliminary study on Unsupervised Term Discovery (UTD). In text-to-image generation, it denotes a curated dataset of over 10K structured supervision instances used to train CoCo (Code-as-CoT), a framework in which a model generates executable code, renders a deterministic draft image, and then performs draft-guided refinement to obtain the final image (Havard et al., 2017, Li et al., 9 Mar 2026).
1. Disambiguation and naming
The supplied literature suggests that the string “CoCo-10K” is polysemous rather than canonical. Its two exact uses are dataset-specific and arise in unrelated technical domains.
| Context | Meaning | Supervision form |
|---|---|---|
| Speech-COCO | 10-hour / 10,000-spoken-caption subset | speech aligned to image and caption text |
| CoCo (Code-as-CoT) | curated dataset of over 10K structured supervision instances | Text–Code pairs and Text–Draft Image–Final Image triplets |
The first usage is tied to the augmentation of MSCOCO with synthetic spoken captions and is explicitly a small, early-use subset of a much larger corpus. The second usage is tied to a two-stage T2I pipeline and is explicitly not a generic image-caption dataset; its supervision targets executable structure and draft-preserving visual refinement. A central source of confusion is that both usages inherit the surface form “CoCo-10K” while differing in modality, annotation granularity, and downstream task family.
2. CoCo-10K as a subset of Speech-COCO
Within Speech-COCO, CoCo-10K is the subset of Speech-COCO used in the preliminary study. The paper describes it as about 10 hours of speech and about 10,000 spoken captions, and emphasizes that it is not a separate dataset but a slice of Speech-COCO chosen for initial experimentation on unsupervised term discovery. Its role is to provide a tractable scale for early multimodal and unsupervised speech-learning experiments before scaling to the full corpus (Havard et al., 2017).
Speech-COCO itself is an augmentation of MSCOCO in which speech is added to image and text. The construction starts from the MSCOCO training and validation sets; each image has at least five human-written captions, and those captions are synthesized into spoken utterances using Voxygen TTS. The resulting corpus contains 123,287 images, 616,767 captions, and more than 600 hours of speech overall. The paper reports 8 voices—4 UK voices (Paul, Elizabeth, Judith, Bronwen) and 4 US voices (Phil, Bruce, Amanda, Jenny)—and notes that the synthesis is corpus-based concatenative synthesis, not parametric synthesis. For the full corpus, the reported average caption length is 10.79 tokens and the average WAV duration is 3.52 seconds.
In this setting, CoCo-10K functions as a proof-of-concept subset. The paper’s stated rationale is not merely computational convenience. The subset is intended to test whether the corpus is already hard enough for discovery, whether it can serve as a first benchmark for multimodal learning, and whether visually grounded synthetic speech can support speech unit discovery.
3. Annotation regime, perturbation, and UTD evaluation in the Speech-COCO setting
A defining property of Speech-COCO, inherited by CoCo-10K, is that alignment is not produced by forced alignment after the fact. Instead, the annotations are created during TTS synthesis. Each spoken-caption WAV file is paired with a JSON file containing metadata and timing information with the structure duration, speaker, synthesisedCaption, timecode, speed, wavFilename, captionID, imgID, and disfluency. The timecode annotations are available at word level, syllable level, and phoneme level, and the accompanying Python tooling can parse these annotations and convert them to Praat TextGrid format (Havard et al., 2017).
The corpus introduces two explicit forms of variability. First, speed perturbation is applied using Sox tempo, with values 1.1, 0.9, and 1.0, each chosen with equal probability; this is intended to introduce intra-speaker variability. Second, disfluencies are inserted to make the synthesized captions sound more natural. The fillers listed are “um”, “uh”, “er”, “huh”, “oh”, and “ah”. Each caption has a 0.3 probability of receiving a filler, and the insertion position is beginning, middle, or end with equal probability.
The paper’s preliminary experiment uses CoCo-10K for UTD with ZRTools, which applies segmental dynamic time warping (SDTW) to find acoustically similar segments and then uses graph clustering to form discovered unit clusters. Evaluation uses the Term Discovery Evaluation (TDE) toolkit, which reports matching, clustering, and parsing, together with precision , recall , F-score , NED, and coverage. A generic F-score is given as
The authors conclude that CoCo-10K is challenging for UTD because precision is low, recall is even lower, and clusters often reflect same-speaker similarity rather than true lexical identity. At the same time, the system does recover useful recurrent patterns such as “a man riding”, “fire hydrant”, “skiing down”, and “a bunch of bananas.”
4. CoCo-10K as a structured supervision corpus for Code-as-CoT
In the CoCo (Code-as-CoT) framework, CoCo-10K is a curated dataset of over 10K structured supervision instances. It is explicitly not a generic image-caption dataset. Instead, it is designed to supervise two distinct capabilities: structured draft construction via code generation and corrective visual refinement from draft to final image. Its two reported supervision formats are Text–Code pairs and Text–Draft Image–Final Image triplets (Li et al., 9 Mar 2026).
The dataset was created in response to two concrete failure modes observed in a zero-shot pilot study on Bagel. First, code generation was unreliable because the model often produced non-executable code, which prevented draft rendering. Second, drafts were ignored during refinement because the model often generated a new image instead of editing the draft. CoCo-10K therefore provides explicit training signal for producing valid, executable code and for learning to preserve and refine a rendered draft rather than overwrite it.
The resulting generation pipeline is summarized as
where is the text prompt, is executable code, is the draft image produced by sandbox execution, and is the final image after refinement. The paper treats the sandbox as important because it makes the reasoning process safe, deterministic, and verifiable.
5. Construction pipeline, training role, and empirical effects in the Code-as-CoT setting
CoCo-10K is built through two pipelines: an Editing Dataset and a Synthesis Dataset. The Editing Dataset is derived from StructVisuals and contains structured chart images with an A-Image and a B-Image. The corrections involve changing data values, fixing labels, or adjusting formatting, while keeping the overall chart structure intact. The Synthesis Dataset is aimed at structured visual generation. Its automated process synthesizes prompts for scientific concepts, charts, posters, infographics, annotated diagrams, and other text-intensive visual formats; uses Gemini-3-Pro to generate code specifying visual layout, object positions, textual placement, and structural organization; executes that code in a sandboxed environment to produce the draft A-Image; and then uses Nano Banana to refine the draft into the final B-Image (Li et al., 9 Mar 2026).
The paper states that supervised fine-tuning begins from Bagel and uses token-level cross-entropy loss on code and MSE loss on the VAE tokens for the final image. Training proceeds in two phases: text-to-code fine-tuning first, followed by full-parameter fine-tuning for 16K steps. The authors also report that the collected data are organized exactly around the pipeline’s intermediate representations, which is the key difference from ordinary T2I datasets that pair only text with a target image.
The empirical effect of this dataset is reported indirectly through the performance of the full CoCo system. On StructT2IBench, CoCo achieves 73.52% overall accuracy, compared to 49.58% for GPT-Image, and the abstract reports improvements of +68.83% over direct generation. The abstract also reports improvements of +54.8% on OneIG-Bench and +41.23% on LongText-Bench. On text-heavy benchmarks, the paper reports 0.853 overall on OneIG-Bench and 0.754 overall on LongText-Bench. The ablations further attribute a central role to the dataset design: without CoCo-style training, Bagel achieves only a 9.06% compilation success rate on LongText-Bench, whereas CoCo fine-tuning reaches 100% compilation success. The mixture-ratio study reports that a small amount of Text–Code supervision is sufficient to induce structured reasoning, while the bulk of training is more effective when allocated to draft-to-final refinement.
6. Non-equivalent usages in adjacent literatures
Several nearby papers in the supplied literature are relevant precisely because they do not define a dataset literally named CoCo-10K. The foundational MS COCO paper defines Microsoft Common Objects in COntext and its official train/validation/test releases, but it does not mention a subset explicitly named “CoCo-10K”; it also reports that the dataset includes five written captions per image, which later enabled Speech-COCO-style augmentation (Lin et al., 2014).
Likewise, COCONut modernizes COCO segmentation and introduces a dataset with 383K images and more than 5.18M panoptic masks, but the paper states that it does not define a dataset called “CoCo-10K” exactly. Its closest named smaller-scale subsets are relabeled COCO-val with 5K images, 67K masks and COCONut-val with 25K images, 437K masks (Deng et al., 2024). COCO-ReM is similarly COCO-derived and manually verified, but the paper states that it does not define anything named “CoCo-10K”; its validation split contains 40,689 instances, not a 10K-branded subset (Singh et al., 2024).
Outside COCO-style vision benchmarks, the label can drift further. In long-sequence recommendation, the Douyin paper states that CoCo-10K is not a benchmark name, dataset, or separate model/system; rather, the paper concerns a 10k-sequence production setting (Guan et al., 8 Nov 2025). In KID readout electronics, the exact term is not named explicitly, but the paper is described as addressing the same 10 mW/pixel milestone that “CoCo-10K” appears to denote in that community. This suggests an informal detector-readout usage centered on a 10,000-pixel-class kinetic inductance detector readout target rather than a COCO-derived dataset (Sinclair et al., 2023).
The practical implication is that CoCo-10K should not be read as a stable universal identifier. In the supplied literature, it denotes either a pilot-scale visually grounded speech subset or a structured supervision corpus for code-driven T2I, while several adjacent papers use related COCO naming, 10K-sized regimes, or 10K-scale milestones without introducing an exact dataset of that name.