---
title: 'CogIP-Bench: LLM Code Interpreter Evaluation'
url: https://www.emergentmind.com/topics/cogip-bench
type: topic
---

# CogIP-Bench: LLM Code Interpreter Evaluation

CogIP-Bench (CIBench) is an evaluation framework for assessing the ability of large language models (LLMs) to invoke and reason with a built-in code interpreter plugin while solving realistic, multi-step data-science tasks. CogIP-Bench defines a task suite and metrics for process and output quality, enables systematic comparison of proprietary and open-source LLMs, and provides actionable insights for advancing tool-use agents in data science workflows [2407.10499].

## 1. Formal Framework and Core Metrics

Let $T = \{T_1,\ldots,T_N\}$ denote the set of $N$ multi-turn data-science tasks (notebooks), $M$ the set of five evaluation metrics, and $E = \{e_{\text{end2end}}, e_{\text{oracle}}\}$ the two evaluation modes. CogIP-Bench is thus defined as $(T, M, E)$.

**Output-Oriented Metrics:**

- **Numeric Accuracy:** 
  $$
  A_{\text{num}} = \frac{(\text{Number of correctly predicted numeric answers})}{(\text{Total number of numeric questions})}
  $$
- **Text Score (ROUGE-L):**
  $$
  A_{\text{text}} = \text{ROUGE-L}(\text{predicted report}, \text{ground-truth report})
  $$
- **Visualization Score (SSIM):**
  $$
  A_{\text{vis}} = \text{SSIM}(\text{generated plot}, \text{reference plot})
  $$

**Process-Oriented Metrics:**

- **Tool Call Rate:**
  $$
  R_{\text{tool}} = \frac{(\#~\text{“run code” commands})}{(\#~\text{questions})}
  $$
- **Executable Rate:**
  $$
  R_{\text{exe}} = \frac{(\#~\text{code runs error-free})}{(\#~\text{code submissions})}
  $$

**Overall Score:**
$$
A_{\text{avg}} = \frac{A_{\text{num}} + A_{\text{text}} + A_{\text{vis}}}{3}
$$

These metrics enable multifaceted assessment of LLM tool-use beyond conventional code correctness, capturing not only what answer is produced, but also the interpretability of outputs and robustness of the code execution process.

## 2. Dataset Construction and Scope

The CogIP-Bench dataset comprises 234 multi-step Jupyter-style notebooks with over 1,900 questions, encompassing seven task categories. Construction proceeds through the following phases:

- **Module and Topic Selection:** Ten core Python libraries—pandas, matplotlib, seaborn, scikit-learn, PyTorch, TensorFlow, LightGBM, nltk, opencv-python, scipy—are selected. For each, GPT-4 proposes approximately 50 representative topics.
- **LLM–Human Cooperative Generation:** 
  - GPT-4 generates notebooks (10–15 steps, explicit parameters, external dataset references).
  - Human authors provide feedback, refine prompts (remove file-IO, ensure reproducibility).
  - Recurring patterns are distilled into templates; new datasets are slotted in (GPT-4–synthesized/recently published data).
  - Manual quality control ensures user-authentic prompts, step runtimes under 1 minute, dataset sizes under 50 MB, and precise ground-truth checks.
- **Session Structure:** Each notebook is a multi-turn, interactive IPython session. Downstream questions depend on earlier cell outputs, simulating realistic, stateful code-interpreter usage across a data science workflow.

This methodology yields a comprehensive, diverse set of multi-turn problems, emphasizing complex, authentic tool-use scenarios.

## 3. Evaluation Modes and Protocols

CogIP-Bench implements the ReAct protocol, alternating between “Reasoning” (model thinks and plans) and “Action” (model submits code). Models are permitted up to three self-debug attempts per question.

- **End-to-End Mode ($e_{\text{end2end}}$):**
  - The model receives only the prompt and its prior context.
  - Must autonomously decide when to invoke the interpreter, interpret errors, debug, and proceed without external intervention.

- **Oracle Mode ($e_{\text{oracle}}$):**
  - On failure (code error or wrong result), ground-truth code and reasoning are injected into the context as an in-context example.
  - Simulates interactive human assistance or few-shot demonstration for downstream tasks.

These modes distinguish autonomous problem-solving from scenarios where corrective feedback is available, enabling analysis of both raw and guidance-augmented LLM performance.

## 4. Experimental Setup

**Model Coverage:**

| Model Type         | Examples                                              | Parameter Scales              |
|--------------------|------------------------------------------------------|-------------------------------|
| Proprietary        | GPT-4-1106-preview, GPT-4o                           | Not specified                 |
| Open-Source        | Llama-2 (7B–70B), Llama-3 (8B, 70B), Vicuna (7B–13B) | 7B–72B                        |
|                    | Qwen (7B–72B), InternLM2 (7B, 20B), Yi (6B, 34B)     |                               |
| Code-oriented      | Code Llama, Codex-style variants                     |                               |
| Others             | DeepSeek, ChatGLM3, Baichuan 2                       | See paper for full list       |

**Evaluation Platform and Environment:**

- OpenCompass evaluation harness.
- Python 3.10, with explicit module versions: pandas 1.5.3, matplotlib 3.7.2, seaborn 0.13.0, scikit-learn 1.2.1, PyTorch 1.13.1, TensorFlow 2.14.0, LightGBM 4.1.0, nltk 3.8, opencv-python 4.8.1.78, scipy 1.11.2.
- Deterministic decoding ($\text{temperature}=0$), up to three self-debug trials per query.
- Tasks vary in complexity: easy (<2 steps), medium (2–4), hard (>4).

This methodological rigor ensures reliable, reproducible assessment across diverse model architectures and parameterizations.

## 5. Key Results and Analytical Findings

### Quantitative Performance

- **Proprietary Models:** 
  - **GPT-4-1106-preview:** $A_{\text{num}}\approx77.8\%$, $A_{\text{text}}\approx78.9\%$, $A_{\text{vis}}\approx64.0\%$ in $e_{\text{end2end}}$; $A_{\text{avg}}\approx75.5\%$ in $e_{\text{oracle}}$.
  - **GPT-4o:** $A_{\text{avg}}\approx74.5\%$.
- **Top Open-Source:** Best models (Llama-3-70B-Instruct, Qwen-72B) achieve $A_{\text{avg}}\approx65\%$.
- **Medium-size Models:** Llama-2-7B, Vicuna-7B attain $A_{\text{avg}}\approx 20$–$35\%$.

### Behavior and Robustness

- **Process Metrics:** Most models invoke the interpreter in over 90% of queries. Executable rates: proprietary models $>95\%$; open-source, $45$–$90\%$.
- **Task Difficulty:** On “hard” (>4-step) tasks, average scores decrease by 10–25 points relative to “easy” tasks, indicating decreased robustness in extended workflows.
- **Module Performance:** High scores on SciPy (mathematics/statistics) and simple visualization; near-zero on advanced modeling (PyTorch, TensorFlow).
- **Debugging:** Allowing two to three self-debug trials yields +5–10% increases in $R_{\text{exe}}$, $A_{\text{num}}$, $A_{\text{vis}}$ for many models, indicating partial self-repair capability.
- **Cross-Benchmark Correlation:** CIBench scores correlate strongly ($r>0.7$) with GSM8K, HumanEval, and BBH metrics, suggesting generalized gains in reasoning and coding transfer to tool-use scenarios.
- **Language Sensitivity:** Most models lose 2–5 points in $A_{\text{avg}}$ when inputs are translated to Chinese; larger degradations occur for some models (e.g., DeepSeek, Qwen), indicating cross-linguistic sensitivity.

## 6. Insights and Recommendations

Analysis of CogIP-Bench results yields several key recommendations:

- **Error-Correction Training:** Incorporate explicit error-parsing and bug-repair capabilities to enhance LLM autonomy following interpreter failures.
- **Multi-Turn Reasoning:** Emphasize architectural support for long-term memory and contextual state tracking in interactive, sequential tool-use.
- **Module-Specific Proficiency:** Target specialized fine-tuning or retrieval-augmented approaches for advanced frameworks (e.g., PyTorch, TensorFlow), as current LLMs—including state-of-the-art—underperform markedly.
- **Metric Extensions:** Develop output-side metrics capturing non-deterministic behaviors (e.g., stochastic model training outcomes) and increase robustness of visualization similarity measures.
- **Language Robustness:** Enhance multilingual pipeline fidelity to maintain performance in non-English data science workflows.

A plausible implication is that future advances in LLM tool-use proficiency hinge not only on scaling and general code competence, but also on dedicated architecture, training, and pre/post-processing tailored to the distinctive demands of interactive, multi-framework data science sessions.

## 7. Significance for LLM Agent Evaluation

CogIP-Bench provides a large-scale, multi-turn, real-world benchmark that exposes the performance gap between proprietary and open-source LLMs in realistic data science scenarios, especially for advanced modeling workflows. Its joint use of process- and output-oriented metrics, under two evaluation modes, offers precise analysis of both autonomous and guided tool-use proficiency. By making stateful, multi-turn problem-solving and comprehensive metrication central, CogIP-Bench sets a new standard for evaluation, facilitating both progress assessment and diagnosis in the development of code-interpreter-empowered agents [2407.10499].

Source: https://www.emergentmind.com/topics/cogip-bench