CookBench: Embodied Long-Horizon Cooking Benchmark

• CookBench is a long-horizon embodied planning benchmark that tests agents’ ability to interpret rich user instructions and execute detailed cooking procedures in a realistic simulated kitchen.
• It employs a two-stage pipeline—intention recognition followed by embodied interaction—to separate natural language understanding from fine-grained physical execution.
• The benchmark’s extensive simulation, realistic physics, and multi-step action sequences highlight current AI gaps in persistent state tracking and spatial reasoning.

CookBench is a long-horizon embodied planning benchmark for complex cooking scenarios that evaluates whether an agent can both interpret rich user intent and execute the resulting cooking procedure through fine-grained physical interaction in a high-fidelity simulated kitchen. Built on a Unity-based environment derived from Cooking Simulator, it frames cooking as a demanding testbed for embodied AI because recipes combine long action sequences, tool- and ingredient-specific affordances, irreversible transformations, temporal constraints, and realistic physics. The benchmark is organized as a two-stage pipeline—Intention Recognition and Embodied Interaction—and is explicitly designed to expose the weaknesses of current LLMs and Vision-LLMs on tasks whose average execution length is around 120 atomic actions (Cai et al., 5 Aug 2025).

1. Research Scope and Motivation

CookBench is motivated by a specific gap in embodied planning research: many established benchmarks emphasize short-horizon tasks, coarse-grained action primitives, or simplified interaction structure, whereas cooking naturally requires long-horizon, compositional, and physically grounded planning (Cai et al., 5 Aug 2025). In the comparison developed around the benchmark, prior environments such as ALFRED, TEACh, and VirtualHome are described as operating mostly in the short- to medium-horizon regime, often around 6–25 high-level steps, with ALFRED averaging about 12. CookBench instead targets a regime in which the average task sequence is around 120 atomic actions.

Cooking is used not merely as a thematic wrapper but as a domain with unusually strong structural properties for planning. Recipes decompose into interdependent sub-tasks; they involve temporal and parallel constraints; they require reasoning about tool semantics, ingredient states, and quantities; and they include irreversible transformations such as cutting, heating, mixing, and pouring. This makes error recovery difficult and turns local mistakes into long-range failures. A plausible implication is that CookBench is intended less as a benchmark for isolated manipulation success than as a benchmark for persistent state tracking, long-range dependency management, and strategic sequencing under realistic physical consequences.

The benchmark also separates natural-language understanding from physical execution. That separation is important because failure may arise either from incorrect goal parsing or from inability to ground a correct plan in the environment. By isolating these stages, CookBench supports diagnosis of whether a system fails at understanding what should be cooked, at planning how to cook it, or at carrying out the required sequence in a visually grounded setting.

2. Environment, Assets, and Recipe Knowledge Base

CookBench is implemented on top of Cooking Simulator in Unity and inherits a high-fidelity kitchen environment with photorealistic rendering and physics-based interactions (Cai et al., 5 Aug 2025). The simulator includes rigid-body dynamics, collisions, continuous pouring, and cooking heat transfer, so the environment is not merely symbolic. Actions such as dropping an item, spilling a liquid, or overcooking food have persistent effects on later execution.

The environment is populated with substantial content. The reported asset inventory includes more than 228 food items, 132 liquids, 70 spices, and 40+ tools. Tasks are instantiated in the sandbox mode of the “Classic” kitchen, while appearance and layout can vary through 10 scene themes and 10 festival decorations. The simulator also provides built-in scoring for final dishes, based on ingredient correctness, cooking methods, and final states.

At the level of domain knowledge, CookBench contains a recipe base of 131 dishes represented in compact, semi-structured procedural text. Each dish specifies ingredients, quantities, methods, and ordering constraints. These dishes are categorized by preparation difficulty into 31 simple dishes, 72 medium dishes, and 28 hard dishes. They are also organized by cuisine or regional style, including French/Mediterranean, American, Central/Eastern/Other European, Italian, Asian/others, and Chinese. The benchmark includes 131 single-dish scenarios and 4,446 multi-dish combinations derived from that recipe set.

The benchmark’s notion of long horizon is therefore multidimensional. It includes not only long action sequences, but also multi-dish composition, branching and partially parallel workflows, and continuous state changes that accumulate over time. This suggests that CookBench should be read as a benchmark for integrated task structure rather than for single-episode execution length alone.

3. Two-Stage Benchmark Design

The core task in CookBench is designed as a two-stage process: Intention Recognition followed by Embodied Interaction (Cai et al., 5 Aug 2025). This decomposition is central to the benchmark’s epistemic structure.

In Intention Recognition, the agent must infer the user’s final cooking goal from natural-language instructions. The associated dataset, CookBench-IR, contains about 14,394 bilingual instructions, roughly 7,197 per language, spanning both single-dish and multi-dish settings. For single-dish tasks, instructions are divided into Direct Instructions, Interference Instructions, and Ambiguous Instructions. Direct Instructions explicitly specify the dish. Interference Instructions contain multiple options, conflicting information, or logical distractors. Ambiguous Instructions omit the dish name and require inference from ingredients, taste preferences, health goals, or regional style. For multi-dish tasks, the benchmark includes commands involving 4, 5, or 6 dishes, with about 1,482 instances per scenario, and distinguishes batch orders from sequential orders.

The output of Intention Recognition is structured. For single-dish input, the model predicts the target dish name. For multi-dish input, it predicts an ordered list of dishes. Evaluation uses strict exact-match accuracy: a prediction is correct only if all dish names and their order exactly match the ground truth. This exactness makes ordering errors first-class failures rather than partial credit cases.

Embodied Interaction begins once the structured goal has been identified. The agent must interpret the corresponding recipe or recipes, plan a long sequence of actions, and execute that sequence in the simulator using visual feedback and static scene knowledge. The benchmark description treats this stage conceptually as a long-horizon MDP-like setting whose state includes object positions, ingredient and tool states, agent pose, timer states, and other environment variables. The action space is the benchmark’s Atomic Action Library, and episode-level performance is determined by dish score and completion time.

A recurrent misconception is to treat CookBench as if it were purely a language-understanding benchmark because of the prominence of CookBench-IR. That is incomplete. The benchmark’s defining claim is that intention recognition is only the first stage; the decisive challenge is whether the recognized intent can be translated into successful, physically grounded execution.

4. Action Granularity, Unified API, and Spatial Grounding

CookBench refines action granularity to a spatial level that retains crucial operational information while abstracting away low-level robotic control (Cai et al., 5 Aug 2025). Its Atomic Action Library comprises about 35 primitives, including navigation and camera control actions, object interactions such as Pick Up and Put Down, food-processing actions such as Cut, Stir / Mix, Crack Open, and Separate, liquid and cookware operations such as Pour and Flip Over, and appliance control actions such as Turn On Appliance, Start Heating, and Add Time. Actions are parameterized. Examples given in the benchmark description include calls such as move_vertical(distance_pixels=-300), pour(liquid="olive oil", amount_ml=50, target="pot"), and cut(target="lemon", cut_type="pieces", piece_weight=16).

This design is deliberately intermediate in abstraction. It is not a motor-control benchmark operating at the level of torques or joint angles, but it is also not a benchmark of purely symbolic macros such as “SliceObject” or “GoToLocation.” Navigation, viewpoint adjustment, amounts, distances, and targets must still be chosen explicitly. That choice makes alignment and reachability significant sources of error.

The unified API encapsulates both macro-level and fine-grained operations. Macro-level functions include placing orders for missing ingredients and purchasing items. Fine-grained embodied actions cover physical interaction with tools, ingredients, and appliances. The intention is to let researchers focus on planning and decision-making while preserving a realistic interface to the environment.

Observation and grounding are likewise structured. The agent receives RGB views as its primary perceptual input. In the benchmark’s demonstration setup, a Vision-LLM continually analyzes images to report whether an action succeeded or failed and to detect anomalies such as being stuck or dropping an item. Static scene information is also exposed through a curated world map describing objects, tools, weights, initial relative positions, and supported operations. In addition, the benchmark uses a spatial knowledge graph derived from 3D semantic maps reconstructed from RGB through VGGT and manually annotated into more than 70 semantic categories. This graph provides object identities, subparts, distances, and spatial relations.

A plausible implication is that CookBench’s API makes hierarchical agent architectures especially natural: LLMs can plan over recipes and goals, while specialized modules handle geometric alignment, navigation recovery, and perception-driven verification.

5. Evaluation Protocol and Empirical Findings

CookBench evaluates the two stages separately and reports a substantial gap between current foundation models and human performance (Cai et al., 5 Aug 2025). For Intention Recognition, the benchmark tests open-source and closed-source LLMs including the Qwen3 family, DeepSeek-V3, DeepSeek-R1, GPT-4.1, and Gemini-2.5-pro. Single-dish English performance is high for the strongest systems, with Gemini-2.5-pro at 96.44%, GPT-4.1 at about 96.69%, DeepSeek-R1 at about 94.58%, and Qwen3-235B at about 93.17%. Single-dish Chinese results are lower, with Gemini-2.5-pro at 92.04% and DeepSeek-R1 at 90.77%, reflecting deliberately harder Chinese instructions. Multi-dish performance drops sharply as the number of dishes increases. In English 6-dish settings, Gemini-2.5-pro is about 55.56%, GPT-4.1 about 37.29%, DeepSeek-R1 about 49.83%, and Qwen3-235B about 44.95%. Overall averages are reported as about 72.91% for Gemini-2.5-pro, 65.78% for DeepSeek-R1, 60.22% for GPT-4.1, and 59.11% for Qwen3-235B.

For Embodied Interaction, the benchmark reports that no model currently can autonomously solve the full visual planning plus control problem. The authors therefore introduce a human-in-the-loop protocol. In that setup, GPT-4.1 functions as planner, GPT-4o serves as a VLM for perception and partial planning, and human intervention is restricted to eight atomic commands: four navigation actions and four camera rotation actions. Human experts achieve perfect scores of 5/5 on all 131 single-dish tasks. By contrast, the human-in-the-loop agent’s average scores remain low across cuisine categories: about 0.49 for French/Mediterranean, 0.43 for American, 0.33 for Central/Eastern/Other European, 0.32 for Italian, 0.70 for Asian/Others, and 0.75 for Chinese.

The benchmark’s failure analysis is as important as its raw numbers. Common errors include navigation failures, reachability and proximity failures, difficulty with multi-source reasoning, slow visual verification loops, and inability to perform fine-grained spatial interaction robustly. The system may correctly infer a high-level plan, such as ordering an unavailable ingredient or decomposing a recipe into sub-tasks, but still fail on the spatial details required to execute even simple locomotion and alignment actions. This result supports the benchmark’s central thesis that high-level planning competence does not yet transfer reliably into grounded long-horizon execution.

Task Completion Score in Embodied Interaction is computed on a 0–5 scale by the simulator and depends on ingredient correctness and quantity, correct application of methods, and final states such as cooking level and temperature. Completion time is also recorded. Together, these metrics make CookBench simultaneously a correctness benchmark and an efficiency benchmark, although its primary analytical focus is on correctness under long-horizon embodied complexity rather than on makespan optimization alone.

6. Position Within the Benchmark Landscape and Research Trajectory

CookBench occupies a specific position among embodied AI benchmarks (Cai et al., 5 Aug 2025). Relative to short-horizon manipulation benchmarks such as BEHAVIOR and ManiSkill2, it shifts emphasis away from low-level control and toward strategic, multi-step planning and multi-task scheduling. Relative to benchmarks such as ALFRED, ReALFRED, TEACh, VirtualHome, ALFWorld, and ET-Plan-bench, it substantially increases horizon length, action granularity, and physical realism. Its distinguishing characteristics are the longest sequence length in the comparison, continuous state changes, fine manipulation granularity, and a knowledge requirement spanning common sense, domain knowledge, and tool semantics.

A second misconception is to interpret CookBench as a benchmark whose primary contribution is time-efficiency evaluation. Later work on ParaCook explicitly contrasts itself with CookBench on this point, describing CookBench and Collab-Overcooked as richer multi-agent, long-horizon cooking tasks that are “too comprehensive to isolate time efficiency evaluation” and that do not treat time efficiency as a primary metric (Zhang et al., 13 Oct 2025). This comparison clarifies CookBench’s niche: it is a broad embodied planning benchmark centered on realistic long-horizon execution, not a benchmark specialized for parallel scheduling or makespan minimization.

The benchmark is presented as a platform for future work rather than as a solved task. The research directions identified around it include hierarchical and hybrid planning, structured memory and state tracking across 100+ steps, language-grounded control, and multimodal integration that combines LLMs, VLMs, and specialized controllers. Because the benchmark exposes both intention recognition and execution gaps, it also supports modular progress tracking: improvements may come from better bilingual goal parsing, better recipe decomposition, more reliable spatial reasoning, or faster perception-action feedback.

CookBench’s broader significance lies in its attempt to operationalize everyday embodied intelligence in a domain where linguistic ambiguity, procedural structure, and physical interaction are inseparable. In that sense, it is less a cooking benchmark in the narrow sense than a standardized stress test for whether embodied agents can bridge rich natural language, long-horizon planning, and realistic action in a single environment. The full benchmark is intended to be open-sourced to facilitate that research trajectory (Cai et al., 5 Aug 2025).