CogIP-Bench (CIBench) Evaluation
- CogIP-Bench (CIBench) is a benchmark that evaluates interactive code interpreter use in realistic, notebook-style data science workflows.
- It employs both end-to-end and oracle evaluation modes to measure tool invocation, execution accuracy, and multi-turn state management.
- Developed through an LLM-human cooperative pipeline, CIBench addresses gaps in autonomous code generation, debugging, and persistent session challenges.
Searching arXiv for CIBench and key related benchmarks to ground the article in current papers. CogIP-Bench, introduced in the paper as CIBench, is a benchmark and evaluation framework for measuring how well LLMs use a code interpreter plugin to solve data science tasks in interactive IPython sessions (Zhang et al., 2024). Its target is not ordinary code generation in isolation, but notebook-style problem solving in which a model must read natural-language instructions, generate Python code, execute it through a code interpreter, observe execution feedback, debug or revise code over multiple attempts, and continue across consecutive, interconnected questions within a persistent session. The framework combines an evaluation dataset with two evaluation modes—end-to-end and oracle—and is designed to assess both autonomous and human-assisted code-interpreter use.
1. Scope and motivation
CIBench was introduced to address a specific gap in the evaluation of LLM-based agents: prior benchmarks often measure mathematical reasoning, one-shot code generation, single-turn tool use, or isolated data science problems, but do not adequately capture realistic code-interpreter utilization in practical workflows (Zhang et al., 2024). The benchmark is explicitly positioned around three missing properties: code interpreter use rather than plain code generation, interactive consecutive workflows rather than single-turn questions, and broad data-science coverage rather than narrow domains. In the paper’s comparison, CIBench is distinctive because it is simultaneously multi-turn, code-interpreter based, and consecutive across sequential questions.
This positioning matters because notebook-based data science work is inherently stateful. Variables, intermediate tables, models, and plots persist across turns, and later questions may depend on the outputs of earlier ones. A benchmark confined to isolated prompts cannot measure compounding error, recovery from failed execution, or the practical ability to continue work inside a live computational session. CIBench therefore treats code-interpreter competence as a compound capability involving instruction following, reasoning, execution, debugging, and sequential context management rather than mere syntax generation.
The paper contrasts this design with datasets such as DS-1000, MINT, CodeGen, and the QwenAgent benchmark, arguing that those benchmarks each miss at least one of the properties CIBench tries to unify. This suggests that CIBench occupies a specialized niche within LLM evaluation: it is a benchmark for interactive notebook workflows rather than a general code benchmark or a generic tool-use benchmark.
2. Dataset construction and design principles
The CIBench dataset is constructed through an LLM-human cooperative pipeline rather than through fully automatic generation or purely manual authoring (Zhang et al., 2024). The paper argues that pure LLM generation risks bias, low diversity, ambiguity, and errors, while pure human construction is expensive and slower. The pipeline therefore combines GPT-4-based generation with expert refinement and manual validation.
The construction process proceeds through several stages. The authors first select widely used Python modules in data science, then prompt GPT-4 to generate topic candidates, and then prompt it again to generate notebook-style tasks. These generated tasks are refined iteratively using prompting, examples of good and bad cases, and constraints on specificity and determinism. Human experts then create template tasks informed by generated tasks and high-quality tutorials from Python libraries. Datasets are collected or synthesized to diversify the task set, and experts manually double-check correctness and practicality using a code interpreter.
A central design target is authenticity. For each sampled module-topic pair, GPT-4 is asked to generate a Jupyter notebook experiment in markdown form, with Python only, approximately 10–15 continuous steps, progression from easy to hard, concise and precise step descriptions, exact parameter names and values, and outputs that are numeric answers, structured output, or visualization. File-saving steps are excluded, and external input data file links are required. The paper states that these tasks are designed to mirror real-world scenarios through notebook-style progression, persistent sequential context, dependence between steps, external data files, varied output types, common data-science libraries, and interactive execution.
The benchmark covers ten common Python modules across several categories:
| Category | Python modules |
|---|---|
| Data Cleaning and Manipulation | Pandas |
| Data Visualization | Matplotlib, Seaborn |
| Modeling | Scikit-learn, PyTorch, TensorFlow, LightGBM |
| Natural Language Processing | NLTK |
| Image Processing | OpenCV-Python |
| Mathematics and Statistics | SciPy |
Topic coverage is expanded systematically. After choosing the modules, the authors prompt GPT-4 to create 50 different topics for each module, intended to cover the majority of the module’s functionality. They then use these topics to guide later task generation. Human refinement is especially important because the authors identify two weaknesses in pure GPT-4 generation: overuse of standard datasets such as Titanic and Iris, and imperfect accuracy in question descriptions or results. Template tasks are therefore created so that task patterns can be adapted across multiple datasets, improving diversity, realism, and reusability.
The paper also describes contamination mitigation. To diversify benchmark datasets, the authors use both GPT-4-generated datasets tailored to templates and latest datasets from the past year. The use of recent datasets is intended to improve authenticity, improve diversity, and reduce the chance that models have already seen the data during pretraining.
3. Corpus organization, session structure, and quality control
CIBench contains 234 tasks and 1900+ questions, organized around consecutive and interactive IPython sessions (Zhang et al., 2024). A task is not an independent one-off prompt; it is a sequence of related questions in a persistent session, where later questions may depend on notebook state produced by earlier ones. This session structure is one of the benchmark’s defining features, because solving later questions may require previously constructed variables, intermediate tables, trained models, or generated plots.
The dataset has three subsets: generation, template, and template_cn. The generation subset contains 960 questions; the template subset contains 470; and template_cn, which is a Chinese translation of the human-created template tasks, contains 470. Across the full benchmark, the appendix breakdown reports 504 numeric-output questions, 116 text-output questions, 788 visualization questions, and 492 “other” questions that only require successful execution without direct output comparison.
Difficulty is defined by the number of interactive steps: Easy up to 2 steps, Medium 2–4 steps, and Hard more than 4 steps. The benchmark is therefore designed not only as a broad data-science suite but also as an explicit probe of interaction depth. This depth-based organization is important because a multi-step notebook workflow is qualitatively different from a single execution request: error propagation, context carryover, and self-correction become central.
Quality control is manual and explicit. Experts use a code interpreter to assure that questions are written from the perspective of a real user, that all required external file paths are informed at the head, that runtime is controlled with each step ideally under 1 minute, that file size is ideally under 50 MB per experiment, and that outputs are valid and unambiguous as ground truth. The appendix states that the authors also serve as annotators for CIBench, specifically in creating template tasks for diverse data science challenges.
This combination of persistent sessions, expert-authored templates, recent datasets, and manual checking is meant to give CIBench a stronger claim to workflow realism than benchmarks based only on standalone prompts. A plausible implication is that CIBench evaluates not only local correctness but also the ability to maintain coherent computational state over extended interactions.
4. Evaluation protocol, environment, and metrics
CIBench follows the ReAct protocol, alternating reasoning traces with code-interpreter invocation, and allows each LLM up to 3 tries per task (Zhang et al., 2024). The benchmark includes two evaluation modes: end-to-end mode and oracle mode. In end-to-end mode, the model must solve the notebook workflow by itself, and each subsequent response depends on its own previous outputs. If early code is wrong, later tasks inherit the flawed context. In oracle mode, when the model fails, it is given the ground-truth code snippet and thought as context for subsequent tasks, simulating human assistance, guided learning, few-shot testing, or in-context learning.
| Mode | Definition |
|---|---|
| End-to-end | Autonomous notebook workflow solving without human help |
| Oracle | Subsequent context is corrected with ground-truth code snippet and thought after failure |
The distinction between these modes is conceptually important. End-to-end mode measures fully autonomous performance under error accumulation, whereas oracle mode measures performance when earlier mistakes are externally corrected. The gap between them reveals how much failure is due to compounding context errors rather than isolated inability on a single step.
CIBench uses both process-oriented and output-oriented metrics. The process-oriented metrics are Tool Call Rate, defined informally as the proportion of instances where the model correctly follows instructions to invoke a code interpreter, and Executable Rate, defined as the percentage of code that is executed without any errors. The output-oriented metrics are Numeric Accuracy, Text Score, and Visualization Score. Text Score uses ROUGE. Visualization Score uses structural similarity (SSIM) between predicted and reference images rather than GPT-4V-based full-benchmark judging, because the authors regard GPT-4V evaluation as expensive and less effective for this setting. They further report a strong correlation between SSIM-based scores and GPT-4V scores on a subset.
The main-results table defines Average as the mean of Num, Text, and Vis in two modes: end-to-end numeric accuracy, end-to-end text score, end-to-end visualization score, oracle numeric accuracy, oracle text score, and oracle visualization score. For “other” tasks, the appendix notes that successful execution is sufficient without direct output comparison.
The evaluation environment is implemented on the OpenCompass platform with a code interpreter using fixed Python library versions. The versions listed are Pandas 1.5.3, Matplotlib 3.7.2, Seaborn 0.13.0, Scikit-learn 1.2.1, PyTorch 1.13.1, TensorFlow 2.14.0, LightGBM 4.1.0, NLTK 3.8, OpenCV-Python 4.8.1.78, and SciPy 1.11.2. The experiments follow ReAct, allow up to 3 attempts, and were run at temperature 0.
5. Experimental results and failure patterns
The paper reports extensive experiments over 24 model entries, although it also notes a small internal inconsistency: the abstract and introduction say 24 LLMs, one experiment section says 19 chat models, and the main results table lists 24 entries (Zhang et al., 2024). The strongest model in the main table is GPT-4-1106-preview with Average 75.5, followed closely by gpt-4o with Average 74.5. The strongest open-source model is Llama-3-70B-Instruct with Average 65.5, which the paper highlights as 10.0% lower than GPT-4-1106-preview. Within scale groups, the best models are Llama-3-8B-Instruct in the approximately 7B group with Average 64.4, InternLM2-20B-Chat in the 13B–20B group with Average 60.6, and Llama-3-70B-Instruct in the 70B group with Average 65.5.
The end-to-end results show very high tool-use process scores for the top proprietary models. GPT-4-1106-preview records Tool 99.8 and Exe 98.6 in end-to-end mode, with Num 77.8, Text 78.9, and Vis 64.0. gpt-4o records Tool 100.0 and Exe 97.2, with Num 74.1, Text 79.0, and Vis 61.1. Among open-source models, Llama-3-70B-Instruct is the strongest end-to-end contender, with Tool 98.0, Exe 91.1, Num 64.9, Text 55.7, and Vis 63.6.
A recurring empirical pattern is that tool invocation is often easy while effective workflow completion is much harder. The paper gives examples such as Qwen-7B-Chat, which has end-to-end Tool 99.8 but Num only 42.3, and Vicuna-7B-v1.5-16k, which has oracle Tool 99.9 but Num only 29.3. This directly supports one of CIBench’s central claims: high Tool Call Rate does not imply strong code-interpreter competence.
Oracle mode outperforms end-to-end mode across all metrics for most models. This suggests that current models benefit substantially from external correction of earlier notebook state and may be more useful in human-in-the-loop workflows than as fully autonomous notebook agents. The paper also finds a typical scaling trend in which larger models generally perform better and rankings are relatively stable within model series.
Performance declines with interaction depth. For example, the difficulty analysis reports Qwen-72B-chat at 60.7 / 66.2 / 50.3 across Easy, Medium, and Hard; DeepSeek-67B-chat at 60.4 / 63.9 / 43.7; Mistral-7B-instruct at 52.3 / 52.4 / 33.0; and Llama-2-70B-chat at 38.1 / 33.9 / 18.6. The benchmark’s category analysis further reports that LLMs perform relatively well on Mathematics and Statistics / SciPy, while open-source LLMs perform poorly on modeling tasks, especially PyTorch and TensorFlow-like modules.
Because the benchmark allows multiple trials, it also probes self-debugging. Increasing the number of allowed trials improves executable rate, numeric accuracy, visualization score, and other metrics, with the biggest gain often appearing when moving from 1 attempt to 2 attempts. This indicates that models can use execution feedback to correct some errors, although not enough to close the gap to the top proprietary systems.
The paper’s manual error analysis of GPT-4-1106-preview classifies failures into four types: Reasoning Errors at 40.4%, Instruction Following Errors at 31.9%, Code Errors at 23.4%, and Hallucination Errors at 4.3%. Reasoning errors are therefore the largest category, which suggests that the bottleneck in notebook workflows is often choosing the correct procedure rather than simply writing executable syntax. The benchmark also reports that most models show a slight drop in Chinese compared to English on the template_cn subset, with especially large drops for DeepSeek-67B-Chat and Qwen-72B-Chat.
6. Relation to prior benchmarks, limitations, and significance
CIBench is positioned against several benchmark families, including math and code benchmarks such as GSM8K, MATH, HumanEval, and MBPP; tool-use benchmarks such as ToolBench, MS-Agent Bench, and MINT; and data-science code benchmarks such as DS-1000, CodeGen, QwenAgent benchmark, and DABench (Zhang et al., 2024). Its claim is not that those datasets are uninformative, but that they omit one or more properties crucial for code-interpreter evaluation: interactive multi-turn structure, actual code interpreter usage, consecutive interdependent tasks, realistic notebook-style workflows, broad data-science module coverage, autonomous and human-assisted evaluation, and combined process and output metrics.
The paper also studies correlation with IFEval, BBH, GSM8K, MATH, HumanEval, and MBPP, finding Pearson coefficients above 0.7, especially with HumanEval and GSM8K. This indicates that CIBench depends strongly on instruction following, reasoning, and coding ability, yet still adds a distinct evaluation dimension because it requires these abilities to operate together in an interactive tool-using workflow. In that sense, CIBench is not a replacement for classical code or reasoning benchmarks but a workflow-level complement to them.
The benchmark claims three main contributions: a new benchmark for agents with code interpreters built through an LLM-human cooperative method and structured as interactive IPython sessions with interconnected questions; two evaluation modes and multiple metrics; and extensive experiments on 24 LLMs showing large gaps between open-source models and GPT-4, especially in modeling modules. The conclusion emphasizes future needs in better error correction based on execution feedback, better understanding of user intent across multiple rounds, and stronger reasoning ability, especially for modeling tasks.
The paper lists two principal limitations. First, CIBench is Python-only, although the same methodology could be extended to other languages. Second, its evaluation metrics have difficulty with some data-science tasks, especially training models with PyTorch and tasks involving randomness. These limitations are substantive rather than incidental: they delimit the benchmark’s current scope and indicate where further evaluation methodology is still needed.
A common misconception is to read CIBench as a standard code-generation benchmark with a notebook wrapper. The reported results argue against that interpretation. High Tool Call Rate does not ensure correct execution, and high Executable Rate does not ensure correct numerical, textual, or visual outcomes. Another misconception is that stronger models merely need more scale; the weak performance of several Llama-2 variants shows that scale alone does not overcome deficits in instruction following and tool-usage behavior. The broader significance of CIBench is therefore methodological: it reframes code-interpreter evaluation around persistent sessions, compounding context, and self-debugging, which are central properties of practical notebook-based data science work.
