CREATOR Case Dataset Overview

• The CREATOR Case Dataset is a specialized benchmark for evaluating LLMs' ability to synthesize reusable Python tools by separating abstract tool creation from concrete decision logic.
• It comprises 2,000 self-contained problems that require crafting custom solutions without defaulting to off-the-shelf libraries, ensuring rigorous model evaluation.
• Empirical results indicate that explicit tool-creation pipelines improve accuracy to around 75%, highlighting the advantages over standard LLM paradigms.

The CREATOR Case Dataset, also referenced as the “Creation Challenge,” is a purpose-built evaluation benchmark introduced within the CREATOR framework for analyzing the abstract and concrete reasoning capabilities of LLMs. The dataset is specifically designed to test LLMs in scenarios requiring the construction of reusable tools rather than direct invocation of off-the-shelf libraries or APIs. It is released as a fixed set of 2,000 test problems, each tailored such that no existing black-box function directly solves the task, and is structured to separate tool creation from downstream decision logic, facilitating precise quantification of LLMs' skills in abstract code synthesis and concrete execution planning (Qian et al., 2023).

1. Structure and Formal Definition

Each entry in the CREATOR Case Dataset is a four-part tuple $(Q, T, C, a)$, where:

• $Q$ is a natural-language problem statement, commonly posing a numerical or algorithmic challenge.
• $T$ is a Python tool (or tools) with a well-defined signature, comprehensive docstring, and executable implementation.
• $C$ is Python code demonstrating how to invoke $T$—possibly with minimal glue logic—to arrive at an answer.
• $a$ is the printed result, serving as the ground-truth numeric answer ($a \in \mathbb{R}$ or integer).

All 2,000 examples are released as a single evaluation-only set, strictly for zero- or few-shot analysis. Demonstrations within prompts are drawn uniformly at random from the dataset for in-context learning; there are no train, validation, or test splits. Each datum is self-contained and includes every artifact needed to both generate a candidate solution and to check correctness via code execution.

2. Construction Methodology and Annotation Workflow

Dataset construction follows a three-step process:

1. Manual Seed Generation: Approximately 100 diverse seed problems are written by the authors, each with a corresponding canonical tool, invocation code, and reference answer.
2. LLM-Assisted Expansion: Each seed is used as a prompt to Text-Davinci-003, eliciting several new variants while ensuring novelty and excluding cases solvable by known standard library calls.
3. Cleanup and Validation: Every example is manually proofread for format consistency, tool function correctness, code executability, and answer validation. No additional human annotation is necessary, as correctness is automatically verifiable via code execution accompanied by the ground-truth printout.

This process yields a dataset where both the tool interfaces and calling conventions are rigorously curated to reflect authentic tool creation and utilization requirements.

3. Problem Archetypes and Query Diversity

The dataset encompasses a broad array of problem types, with 20–30 archetypal templates systematically expanded into approximately 70 variants each. Problem types include:

• Polynomial Fitting / Numerical Analysis: E.g., constructing a custom wrapper around numpy.polyfit for high-degree regression and extrapolation tasks.
• Combinatorics / Custom Counting: Requiring the synthesis of bespoke counting routines (e.g., implementing prime-checking logic).
• Simulation / Iterative Search: Demanding models to craft solvers (e.g., writing a dedicated solve_quadratic rather than using off-the-shelf functions).
• Data-Structure Manipulation: Such as computing areas of convex hulls via ad hoc computational geometry routines.
• Custom API Aggregation: Tasks requiring orchestration around hypothetical or compositional APIs.

A representative example is the degree-5 polynomial regression entry, where the query describes extrapolating epidemic curve data and the canonical tool definition encapsulates fitting and prediction using numpy.polyfit and numpy.polyval.

4. Quantitative Properties and Sampling Protocol

Statistic	Value/Protocol	Comments
Dataset Size $(|D|)$	2,000 entries	Released as a fixed 2K set
Avg. Query Length	$\sim$80 tokens (median $\approx$75)	Substantial natural-language variety
Avg. Tool Definition Length	$Q$025 lines Python (incl. docstring)	Substantial code complexity
Vocabulary Size	$Q$1 3,000$$unique word-pieces</td> <td>Across all queries and docstrings</td> </tr> <tr> <td>Execution Success</td> <td>$$Q$2 under CREATOR pipeline	By design, code is syntactically valid
Diversity	20–30 templates, $Q$370 variants each	Covers core tool-creation scenarios

For few-shot prompting, demonstration sets $Q$4 are sampled uniformly at random, with $Q$5 and $Q$6. There is no use of stratified or class-balanced splits; uniform random selection achieves coverage at inference time.

5. Benchmarking, Evaluation Protocol, and Empirical Results

The dataset is used exclusively as a zero- or few-shot evaluation tool. No fine-tuning or training is performed directly on the Creation Challenge set. During evaluation, a handful of demonstration examples may be included as context, but the model's primary task is to synthesize both the tool definition and the correct tool-calling logic for previously unseen problems.

Empirical results from the CREATOR framework demonstrate substantial gains over baseline LLM paradigms. Specifically, the four-stage pipeline (Creation → Decision → Execution → Rectification) achieves approximately $Q$7 accuracy on the Creation Challenge, compared with approximately $Q$8 for standard LLMs and $Q$9 for the "program-of-thought" (PoT) baseline. These results quantify the impact of explicit tool-creation reasoning combined with concrete execution guidance.

6. Evaluation of Abstract and Concrete Reasoning

The dataset is intentionally structured to dissociate abstract tool creation from concrete decision execution. Tool creation targets the LLM's capacity to formulate minimal, reusable code modules capturing the essential computational structure of a problem. Concrete decision execution probes the model's skill in correctly invoking these tools, managing input/output transformations, and integrating into the solution pipeline.

Because each question is constructed to preclude resolution via a single off-the-shelf library or pre-existing API, success on the Creation Challenge is posited as a proxy for genuine problem-solving ability—minimizing spurious pattern matching or memorization effects. The dataset thus offers controlled experimental conditions for disentangling high-level code abstraction abilities from low-level execution planning.

7. Significance and Applications

The CREATOR Case Dataset provides a rigorous instrument for evaluating LLMs' ability to generalize tool-creation skills. By emphasizing scenarios where tool synthesis, rather than lookup, is necessary, it supports granular assessment of models’ abstract reasoning capacity. The quantifiable performance gains observed when employing pipelines specialized for tool synthesis suggest that disentangling these reasoning stages is a promising avenue for robust LLM development. Given its breadth and rigorously validated structure, the dataset is suitable for benchmarking advances in program synthesis, LLM-based tool construction, and compositional reasoning (Qian et al., 2023).