---
title: 'CraftBench: Dual-Domain Benchmark'
url: https://www.emergentmind.com/topics/craftbench
type: topic
---

# CraftBench: Dual-Domain Benchmark

CraftBench is an ambiguous benchmark name in recent machine-learning literature. In the provided arXiv sources, it denotes at least two distinct evaluation resources: a benchmark for scientific figure generation across multiple figure types and input conditions, and a procedural-video benchmark for furniture crafting and assembly that was later extended into a fact-verification resource called CraftBench-Fact [2605.30611] [2604.25584]. This multiplicity matters because nearby work in crafting, construction, creativity, and Minecraft-inspired evaluation is often lexically similar but not equivalent, and the term therefore has no single canonical referent across all current arXiv usage.

## 1. Nomenclature and referential scope

The clearest documented uses of the name fall into two separate domains. One use appears in scientific-figure generation, where CraftBench is introduced as a benchmark spanning three figure types and four input conditions. The other appears in procedural video understanding, where CraftBench refers to crafting videos with temporally aligned action-level captions and later to a structured factuality extension, CraftBench-Fact [2605.30611] [2604.25584].

This ambiguity is reinforced by several adjacent names that are explicitly not aliases. The paper introducing CreativeBench states that it is about **CreativeBench**, not “CraftBench,” and provides no evidence that “CraftBench” is an alternate name, abbreviation, prior name, or subset of the same project [2603.11863]. CrafText is strongly related at the level of open-ended crafting environments and instruction following, but it is presented as **CrafText**, not CraftBench [2505.11962]. Likewise, BuilderBench, MinePlanner, and the Minecraft builder-dialog benchmark are construction- or planning-oriented resources relevant by theme, not by identity [2510.06288] [2312.12891] [2407.12734].

| Name in source | Domain | Relation to “CraftBench” |
|---|---|---|
| CraftBench | Scientific figure generation | Direct use of the name |
| CraftBench / CraftBench-Fact | Procedural crafting videos | Direct use of the name |
| CreativeBench | Code-generation creativity | Explicitly not shown to be CraftBench |
| CrafText | Craftax instruction following | Related but differently named |
| BuilderBench | Embodied block construction | Adjacent, not equivalent |

A plausible implication is that “CraftBench” should be treated as a homonymous benchmark label whose meaning depends on the paper and domain rather than as a single benchmark family.

## 2. CraftBench in scientific figure generation

In the scientific-figure literature, CraftBench is introduced to evaluate figure generation beyond narrow text-to-image academic-diagram settings. Its defining scope is **3 figure types/styles**—**Academic figures**, **Posters**, and **Infographics**—together with **4 input conditions/tasks**: **Text-to-image**, **Mask-completion**, **Sketch-conditioned generation**, and **Key-element composition**. The benchmark contains **279 samples** in total, with **179** text-to-image samples, **30** mask-completion samples, **40** sketch-conditioned samples, and **30** key-element-composition samples; by figure family, it contains **140 Academic figures**, **109 Posters**, and **30 Infographics** [2605.30611].

| Component | Count |
|---|---:|
| Total samples | 279 |
| Academic figures | 140 |
| Posters | 109 |
| Infographics | 30 |
| Text-to-image | 179 |
| Mask-completion | 30 |
| Sketch-conditioned generation | 40 |
| Key-element composition | 30 |

The benchmark is curated from published papers across **18 research areas**, **award-tier conference posters**, and **research blogs**. The appendix describes a seven-stage quality pipeline: caption keyword filtering, strict content classification, complexity rescoring, alignment verification, first-pass QA, evidence-required QA, and manual review. For reference-conditioned inputs, every sample is reviewed by **three graduate-level annotators** and accepted only on **unanimous agreement**. The paper does **not report train/validation/test splits** for CraftBench; it is presented as an evaluation benchmark of curated samples rather than a supervised training corpus [2605.30611].

Evaluation uses a **referenced VLM-as-judge protocol** with **Gemini 3.5 Flash**. Candidate outputs and human targets are scored independently rather than in side-by-side comparison. For text-to-image, the scored aspects are **content faithfulness**, **readability**, and one style-specific format aspect; for reference-conditioned tasks, the aspects are **content faithfulness**, **readability**, and **input fidelity**. The benchmark-level score is a **lenient win-rate** obtained by mapping verdicts to **Model → 100**, **Tie → 50**, and **Human → 0**, then averaging. In the main reported results, **Crafter (w/ Nano Banana Pro)** achieves **52.30** overall, compared with **29.00** for **PaperBanana (w/ Nano Banana Pro)** and **22.40** for **Nano Banana Pro** alone; under a controlled same-backbone comparison with Nano Banana 2, **Crafter** scores **50.20** versus **28.00** for **PaperBanana** [2605.30611].

Within this literature, CraftBench functions as evidence for cross-type and cross-condition generalization. It is explicitly designed to expose failures of pipelines that perform adequately on narrow academic-diagram text-to-image settings but degrade on sketches, masks, or key-element preservation.

## 3. CraftBench in procedural video understanding and CraftBench-Fact

A distinct use of the name appears in procedural video understanding. Here, CraftBench is a benchmark built around **furniture and utility crafting videos**, with materials such as **wood and metal** and tools such as **saws, drills, clamps, and fasteners**. Videos are selected so that the procedure is **visually demonstrated** by a narrator and the narration provides **step-by-step instructions** aligned with performed actions. The original CraftBench setup is described as containing **action-level, temporally aligned captions** for crafting videos [2604.25584].

The later extension, **CraftBench-Fact**, adds a structured factual layer through three steps: **Clause decomposition**, **Implicit argument augmentation (VIA)**, and **Structured fact annotation**. VIA expands omitted but visually grounded **patient/object**, **tool**, and **location** arguments. From those augmented captions, annotators derive **contextual facts**—grounded predicate-argument relations—and **conceptual facts**—abstract semantic role assignments. The paper gives the example
$$
\mathcal{F}_g^{ctx} = \{\text{stir(soup)}, \text{stir(with spoon)}, \text{stir(in pot)}\}
$$
and
$$
\mathcal{F}_g^{con} = \{\text{Action = stirring}, \text{Ingredient = soup}, \text{Tool = spoon}, \text{Location = pot}\}.
$$
It then uses these structures for dual-layer factuality verification [2604.25584].

The published dataset statistics for CraftBench-Fact are explicit. The **Train** split contains **120 videos**, **1,735 clips**, **2,132 VIA annotations**, **4,468 conceptual facts**, and **3,429 contextual facts**. The **Test** split contains **100 videos**, **1,468 clips**, **1,888 VIA annotations**, **4,197 conceptual facts**, and **3,108 contextual facts**. Overall, the benchmark comprises **220 videos** and **3,203 clips** [2604.25584].

The evaluation framework distinguishes **DualFact-T**, which verifies facts against textual evidence, from **DualFact-V**, which verifies facts against video-grounded evidence. Fact-level support is aggregated into **MultiFactScore**:
$$
\text{MultiFactScore} = \frac{ |\{ f_i \in F : \hat{y}_i = SUPPORTED \}| }{ |F| }.
$$
A central finding is that standard caption metrics obscure omissions and role-level inconsistencies, while video-grounded verification shows that caption-only evaluation can overestimate hallucination by treating visually grounded but task-irrelevant mentions as unsupported. On CraftBench, VIA improves lexical metrics such as **BLEU: 1.17 → 1.66**, **ROUGE: 18.00 → 21.74**, and **SPICE: 7.47 → 11.39**, while fact extraction itself is near-ceiling, indicating that downstream factuality errors are not mainly due to extraction noise [2604.25584].

## 4. Annotation and evaluation paradigms

The two CraftBench usages differ sharply in annotation object, supervision protocol, and evaluation target. The scientific-figure benchmark centers on multimodal generation under conditional inputs; the procedural-video benchmark centers on action-aligned captioning and role-aware factual verification. Their annotation pipelines therefore operationalize different notions of correctness [2605.30611] [2604.25584].

| Aspect | Scientific-figure CraftBench | Procedural CraftBench-Fact |
|---|---|---|
| Primary object | Generated figure image | Procedural caption facts |
| Inputs | Caption, paper context, and optionally mask/sketch/key elements | Video clip and aligned caption |
| Human review | Three graduate-level annotators; unanimous agreement for reference-conditioned samples | Four trained annotators with calibration and adjudication |
| Automatic evaluation | Referenced VLM-as-judge with lenient win-rate | Textual and video-grounded fact verification |
| Output structure | Image-level scores | Conceptual and contextual facts |

The scientific-figure benchmark validates benchmark construction through manual review and validates automatic evaluation through a blind human study. The reported agreement between the automatic judge and majority human verdict is **72%**, with **Cohen’s $\kappa = 0.58$** [2605.30611]. The procedural-video extension reports annotation reliability directly at the structural-label level: **Cohen’s $\kappa = 0.93$** for clause segmentation, **0.87** for VIA, **0.92** for conceptual facts, and **0.97** for contextual facts [2604.25584].

This contrast is methodologically important. One CraftBench treats evaluation as referenced multimodal quality assessment; the other treats evaluation as fact support under textual and video evidence. The name is shared, but the operational semantics of “benchmark performance” are substantially different.

## 5. Related benchmarks and recurrent confusions

Several benchmark families surround the CraftBench name but should be distinguished from it. **CreativeBench** is a benchmark for machine creativity in code generation with subsets **CreativeBench-Combo** and **CreativeBench-Explore**; the paper explicitly states that there is **no textual evidence** that CreativeBench is also called CraftBench [2603.11863]. **CrafText** is a **Craftax-based multimodal benchmark** for instruction following in a dynamic open-ended world, with **3,924 instructions**, **12 scenarios**, and **496 goals**, but it is presented as a related benchmark rather than a CraftBench alias [2505.11962].

In embodied construction, **BuilderBench** targets open-ended block building with a hardware-accelerated simulator, a task suite with **over 42 diverse target structures**, and an emphasis on self-supervised exploration. It is described as strongly relevant to the construction-oriented subset of a hypothetical CraftBench, but not as CraftBench itself [2510.06288]. **MinePlanner** is a planning-oriented benchmark for long-horizon Minecraft tasks compiled into PDDL, with **45 tasks overall** and support for automatically creating propositional and numeric instances; it is relevant as a symbolic planning reference, not as a direct CraftBench definition [2312.12891]. The paper **“A LLM Benchmark based on the Minecraft Builder Dialog Agent Task”** proposes a synthetic Minecraft-inspired spatial benchmark organized around **Absolute Addressing**, **Relative Addressing**, and **Primitive Shapes**; it is explicitly characterized as related background rather than CraftBench [2407.12734].

A separate line of work around the **Craft Assembly Task** is also adjacent rather than identical. **“Component Selection for Craft Assembly Tasks”** formalizes assembly from imperfect components given a single RGB image and available primitive scene objects, using segmentation, template retrieval, pose optimization, primitive simplification, and correspondence search [2407.14001]. **“Prompt2Craft”** later formulates an LLM-centered structured assembly planner for functional craft assemblies, but again does not define CraftBench itself [2512.04568]. These papers are important because they show that “craft” in current literature spans multiple problem classes—scientific illustration, procedural video understanding, assembly from imperfect parts, and open-world instruction following—without a unified benchmark namespace.

## 6. Significance, limitations, and current status

The main significance of CraftBench lies in what the two benchmark lines reveal about benchmarking practice. In scientific figure generation, CraftBench broadens evaluation beyond text-only synthesis and introduces cross-condition testing with masks, sketches, and key-element composition [2605.30611]. In procedural video understanding, CraftBench-Fact shows that fluent captions can remain factually incomplete and that role-sensitive verification can expose omissions, hallucinations, salience errors, and role inconsistencies that lexical overlap metrics do not capture [2604.25584].

Both lines also expose important limitations. The scientific-figure benchmark is relatively small at **279 samples**, is imbalanced toward **academic and poster figures**, and relies on a **closed-source judge**, **Gemini 3.5 Flash** [2605.30611]. The procedural benchmark is currently narrow in domain—**furniture crafting and assembly**—and its factual extension explicitly notes that it does **not model attribute-oriented facts** such as size or spatial properties [2604.25584]. More fundamentally, the shared name obscures substantial differences in task definition, annotation schema, metric design, and intended use.

This suggests that “CraftBench” should always be disambiguated by citation and domain. In current arXiv usage, it is not a single benchmark with stable semantics, but a label attached to at least two distinct research artifacts: one concerned with multimodal scientific figure generation, and one concerned with procedural crafting video understanding and factual verification.

Source: https://www.emergentmind.com/topics/craftbench