CookViz: Benchmark for Cooking Visualization

• CookViz is a publicly available dataset of over 40,000 step-level image–text pairs designed for visualizing sequential cooking processes.
• It supports research in image synthesis and multimodal retrieval by providing curated, temporally coherent cooking instructions paired with images.
• The dataset underpins diffusion-based models such as Chain-of-Cooking to improve cross-modal alignment and ingredient traceability in generated sequences.

CookViz refers to a large-scale, publicly released dataset and associated research ecosystem designed to advance stepwise cooking process visualization. Developed as part of the methodological advances in "Chain-of-Cooking: Cooking Process Visualization via Bidirectional Chain-of-Thought Guidance" (Xu et al., 29 Jul 2025), CookViz enables new forms of multimodal, temporally coherent modeling at the intersection of image synthesis, food analysis, and procedural grounding. Built from over 40,000 carefully cleaned image–text pairs augmented with additional procedural benchmarks, CookViz serves as a substrate for a range of diffusion-based generative models that aim for fine-grained consistency, ingredient traceability, and inter-frame semantic alignment throughout cooking sequences.

1. Dataset Design and Core Statistics

CookViz comprises 40,362 step-level image–text pairs, curated from popular online recipe sources. Each pair consists of a cooking instruction and a corresponding intermediate photograph. Only recipes with at least three steps are included; the total number of recipes, distribution of steps per recipe, and cuisine/instructional diversity are not specified in the reporting publication. Labels, metadata, and schema are primarily limited to the following for each step:

• recipe_id (string)
• step_index (1-based integer)
• instruction_text (string)
• image_path (string)

Dataset cleaning encompasses regular expression-based removal of emojis and spurious characters from instructions, and watermark removal from images using OCR detection and inpainting via Stable Diffusion. Annotation is limited to automated pairing; no manual, stepwise re-labeling or ingredient markup is reported. CookViz is made available for public use, with training and test split at a 9:1 ratio (Xu et al., 29 Jul 2025).

Statistic	Value
Total image–text pairs	40,362
Minimum steps per recipe	3
Training vs. Test split	90% / 10%
Language	English

The dataset provides the first large-scale, publicly available benchmark tailored to the cooking process visualization task, contrasting with earlier datasets such as Cookpad, MIAIS, MM-Res, and RecipeQA, which are either non-public or focused on endproduct/QA rather than stepwise imagery.

2. Data Structure, File Organization, and Access Patterns

CookViz is structured for both scalability and stepwise model training. Recipes are stored in top-level folders ("train", "test"), with each subdirectory named for a single recipe and containing per-step files (e.g., "step_01.txt", "step_01.jpg"). An example folder arrangement is:

CookViz/ ├─ train/ │ ├─ recipe_0001/ │ │ ├─ step_01.jpg │ │ └─ step_01.txt │ └─ ... └─ test/ └─ recipe_0500/...

The minimal schema for each step comprises the recipe identifier, step index, text, and image reference. No explicit provision is made for ingredient, utensil, or region segmentation. A plausible (but not officially specified) record organization extends to original source URLs and potential author data. Loading code in Python is exemplified in the reporting paper, supporting efficient batch access and compatibility with most procedural generation pipelines (Xu et al., 29 Jul 2025).

3. Benchmark Tasks and Evaluation Protocols

CookViz is explicitly constructed to support three primary research directions:

• Step-by-step conditional image generation: Synthesizing realistic images per cooking step, given text instructions.
• Multimodal retrieval: Retrieving relevant images given partial text (or vice versa), supporting procedural search and evaluation.
• Cross-modal alignment: Assessing temporal and semantic consistency between textual steps and image sequences ("semantic consistency over time").

Recommended train/test splits are 90%/10%. No additional validation set is pre-defined, though the 10% training carve out is suggested as a standard protocol. Evaluations reported in the foundational publication utilize several metrics:

• FID (Fréchet Inception Distance): global image realism per generated/real step-pair.
• CLIP-T: CLIP-based recipe/image alignment score.
• CLIP-I: CLIP-based score for frame-to-frame coherency.
• DreamSim: Perceptual sequence coherence.
• Human judgments: “Consistency” (text–image, per step) and “Coherency” (adjacent frames).

No per-object segmentation or ingredient labeling is evaluated within the dataset, but downstream models leveraging CookViz typically implement internal or proxy measures for cross-step consistency and ingredient faithfulness (Xu et al., 29 Jul 2025, Wang et al., 15 Jan 2025, Zhang et al., 3 Dec 2025).

4. Relation to Procedural Image Generation Models

CookViz is deployed as a benchmark and data substrate in several recent multi-step cooking image synthesis frameworks. In "Chain-of-Cooking" (Xu et al., 29 Jul 2025), it underpins a model featuring a Dynamic Patch Selection Module and Bidirectional Chain-of-Thought Guidance, supporting patch-level references across frames for ingredient and action consistency. CookViz is also referenced as a potential resource for evaluation and further development in CookingDiffusion (Wang et al., 15 Jan 2025) and modular systems such as CookAnything (Zhang et al., 3 Dec 2025).

These models generally follow similar evaluation protocols, leveraging CookViz's granular, intermediate-state images to optimize both intra-step semantic fidelity and temporal coherence. Metrics such as FID, CLIP similarity, and human “usability” judgments scaffold comparative analysis.

5. Integration with Generative and Diffusion Architectures

CookViz is specifically suited to drive the training and validation of step-aligned generative frameworks, notably latent diffusion models. For instance:

• CookingDiffusion (Wang et al., 15 Jan 2025) introduces Memory Nets (Text/CLIP Image/Multimodal) to encode prior procedure states, which could be applied directly to CookViz for improved context retention.
• CookAnything (Zhang et al., 3 Dec 2025) employs Step-wise Regional Control, Flexible Rotary Position Embedding, and Cross-Step Consistency Control to ensure per-step disentanglement and ingredient/scene continuity; CookViz provides the per-step, real-world grounding to benchmark these controls.
• VisualChef (Kuzyk et al., 23 Jun 2025), though focused on egocentric video and mask-inpainting, frames its pipeline as directly transferable to a CookViz-like dataset, emphasizing mask-based object grounding for interpretable, context-preserving action depiction.

These methodical architectures utilize CookViz's step-level granularity to learn action-conditioned transformations, step-wise composition, and cross-modal alignment—tasks not feasible with prior, less granular cooking datasets.

6. Known Limitations and Design Considerations

• Recipe and Step Ambiguity: The dataset includes only recipes with at least three steps, but does not publish step distributions or recipe variety information. This may limit statistical modeling of rare or short-form procedures.
• Lack of Explicit Metadata: While image-text correspondence is rigorous, there is no ingredient segmentation, action region markup, or utensil ground truth, leading to reliance on downstream NLP/CV models for fine-grained alignment.
• Watermark Handling: All image watermarks are automatically removed by Stable-Diffusion-based inpainting, introducing a possible domain mismatch (i.e., generated over painted regions) in a minority of samples.
• Licensing and Ethics: The license is not specified within the source publication; users must refer to the (forthcoming) public release to clarify re-distribution, commercial use, and derivative work allowances.

7. Impact, Applications, and Prospects

CookViz establishes a new benchmark and resource for sequential, visually grounded cooking process modeling. Immediate applications include:

• Robust evaluation and improvement of multi-step diffusion-based generative models.
• Benchmarking for temporal consistency, ingredient traceability, and semantic faithfulness.
• Enabling construction of interactive or educational cooking visualization tools, where stepwise image synthesis augments textual or robotic instruction.

Further evolution of the dataset and surrounding ecosystem could support increased annotation granularity (e.g., ingredient masks, utensil bounding boxes), multi-language text, or richer temporal modeling via video.

CookViz thus defines the state of the art for conditional, interpretable, and chronologically coherent cooking visualization, and serves as a foundational resource for the next generation of procedural content generation models (Xu et al., 29 Jul 2025).