---
title: 'MBPP Pro: Advanced Code Generation Benchmark'
url: https://www.emergentmind.com/topics/mbpp-pro
type: topic
---

# MBPP Pro: Advanced Code Generation Benchmark

MBPP Pro is an advanced code generation benchmark designed to evaluate large language models (LLMs) on self-invoking code generation, a task that tests the models’ capabilities in progressive reasoning, problem decomposition, and the correct reuse of generated subroutines. Each MBPP Pro instance extends a standard MBPP problem by adding a more complex “pro” problem that requires reusing the base solution as a subroutine. This design exposes weaknesses in current LLMs’ function-chaining and self-invocation abilities, providing a more rigorous assessment beyond traditional single-function benchmarks [2412.21199].

## 1. Definition and Structure of MBPP Pro

MBPP Pro redefines the Multiple Problems for Python Benchmark (MBPP) by introducing a two-stage, self-invoking code generation paradigm for each benchmark item:

- **Base Problem**: A standard MBPP function-generation task, with a canonical function signature and associated unit tests.
- **Self-invoking Problem**: A new, more complex task necessitating the invocation of the base solution, typically as a subroutine, often requiring multiple calls.
- **Joint Test Suite**: A consolidated suite evaluating both the correctness of the base function and its use within the pro function.

MBPP Pro measures not only the ability to produce correct base implementations but also the capacity to utilize prior outputs as callable elements in larger compositions. For example, the base problem might require counting primes up to $n$, while the pro problem demands applying this function to a list, returning a list of counts via repeated calls to the base function.

## 2. Dataset Construction Pipeline

The MBPP Pro benchmark is produced through a structured, three-stage pipeline:

```python
# Stage 1: Problem generation
for each (base_problem, base_solution) in MBPP:
    prompt = self_invoking_prompt(base_problem, base_solution)
    new_problem, new_solution, test_inputs = DeepseekV2.5(prompt)
    record (base_problem, new_problem, new_solution, test_inputs)

# Stage 2: Solution verification
for each candidate in generated_candidates:
    outputs = run_in_sandbox(candidate.new_solution, candidate.test_inputs)
    if runtime_error(outputs):
        human_fix(candidate.new_solution)

# Stage 3: Test-case assembly
for each candidate in generated_candidates:
    tests = [f"assert {call} == {out}" for call, out in zip(candidate.calls, outputs)]
    human_review_and_refine(tests)
    save final_problem, final_solution, tests
```

In LaTeX-style pseudocode:
\[
\begin{algorithmic}[1]
\For{each base instance $(P_b,S_b)$ in MBPP}
  \State $\,(P_p,S_p,I)\gets \mathrm{DeepseekV2.5}(P_b,S_b)$
  \State $O\gets\textsf{Exec}(S_p,I)$
  \If{$\exists$ error in $O$}
    \State $S_p\gets\textsf{HumanFix}(S_p,O)$
  \EndIf
  \State $\mathcal{T}\gets\bigl\{\mathtt{assert}\;c_i{=}o_i\;\big|\;(c_i,o_i)\in(I,O)\bigr\}$
  \State human\_review($\mathcal{T}$)
  \State store $(P_b,P_p,S_b,S_p,\mathcal{T})$
\EndFor
\end{algorithmic}
\]

This pipeline ensures reproducibility and correctness through both automated generation (using DeepseekV2.5) and subsequent human validation.

## 3. Dataset Statistics and Complexity

MBPP Pro maintains a consistent size with the original MBPP:

| Property                | MBPP            | MBPP Pro                   |
|-------------------------|-----------------|----------------------------|
| Number of problems      | 500             | 500                        |
| Problems per example    | 1               | 2 (base and self-invoking) |
| Test cases per example  | 3–5             | 3–5                        |
| Mean solution lines     | ≈ 7             | ≈ 11–12                    |

- **Solution Length**: Canonical pro solutions increase by approximately 60% in line count relative to base solutions, reflecting the added compositional complexity.
- **IO Size**: The typical MBPP base test involves small lists or single integers, while MBPP Pro test inputs frequently incorporate nested lists or dictionaries, resulting in roughly a 1.5× increase in mean input size.

## 4. Representative Example

A canonical MBPP Pro instance illustrates the benchmark’s compositional demand:

**Base Problem (ID 087):**
- Function: `def count_primes(n: int) -> int:`
- Task: Count primes ≤ n.

**Pro Problem:**
- “Given a list of integers, return a list of counts of primes up to each integer in the list. You must call your `count_primes` function on each element.”

**Canonical Solutions:**
```python
def count_primes(n):
    def is_prime(k):
        if k<2: return False
        for i in range(2,int(k**0.5)+1):
            if k%i==0: return False
        return True
    return sum(1 for i in range(2,n+1) if is_prime(i))

def count_primes_list(nums):
    return [count_primes(x) for x in nums]
```
**Tests:**
```python
assert count_primes_list([10, 5, 1]) == [4, 3, 0]
assert count_primes_list([]) == []
assert count_primes_list([2, 3, 11]) == [1, 1, 5]
```

## 5. Evaluation Metrics and Protocol

The primary evaluation metric is the “pass@k” formula, consistent with HumanEval and MBPP:

\[
\mathrm{pass}@k = 1 - \prod_{i=0}^{k-1}\frac{n-c-i}{n-i}
\]

where $n$ is the number of completions and $c$ is the number of correct generations. Evaluations are performed in both zero-shot and 1-shot settings. For a subset of models, pass@5 and pass@10 are reported ($n=20$, $T=0.2$, top\_p=0.95).

## 6. Model Performance and Error Analysis

Experimental results demonstrate a pronounced decline from MBPP to MBPP Pro, particularly in zero-shot pass@1:

| Model             | MBPP pass@1 | MBPP Pro pass@1 | Absolute Drop |
|-------------------|-------------|-----------------|--------------|
| o1-mini           | 93.9%       | 68.3%           | −25.6 pp     |
| GPT-4o            | 86.8%       | 70.9%           | −15.9 pp     |
| GPT-4-Turbo       | 85.7%       | 69.3%           | −16.4 pp     |
| Claude-3.5-sonnet | 91.0%       | 66.4%           | −24.6 pp     |

Across more than 20 models, the average absolute drop is 10–15 percentage points. Even the best models, which achieve >90% on MBPP, typically fall below 75% on MBPP Pro. 1-shot prompting yields a mean improvement of approximately 10 percentage points for MBPP Pro, but absolute performance remains well below original MBPP levels.

## 7. Failure Modes and Recommendations

MBPP Pro systematically reveals distinct categories of execution failure, summarized below:

| Error Type         | Description                                           |
|--------------------|------------------------------------------------------|
| AssertionError     | Code compiles but fails the test suite               |
| NameError          | Invocation of undefined variable/function            |
| ValueError         | Incorrect number/type of values for unpacking        |
| IndexError         | Out-of-bounds list indexing                          |
| TypeError          | Invalid operand types (e.g., string**int)            |
| OtherError         | KeyError, SyntaxError, ZeroDivisionError, etc.       |

Figure 8 in [2412.21199] shows AssertionErrors are responsible for approximately 50% of MBPP Pro failures, with NameErrors and TypeErrors forming the majority of the remainder. “Chaining errors”—where the base function is correctly solved but incorrectly called—are a failure mode unique to self-invoking tasks.

The authors report that Chain-of-Thought (CoT) prompting produces modest, consistent reductions in AssertionErrors and NameErrors (~10%), and that current instruction-fine-tuning yields only marginal benefit for self-invoking tasks. This suggests an explicit focus on “function-chaining” and progressive reasoning in future data and architectural priors. Dedicated fine-tuning curricula targeting self-invocation and enhanced tracking of previously generated code are specifically recommended. Release of BigCodeBench-Lite Pro provides evidence that these trends generalize beyond MBPP.

MBPP Pro thus provides a reproducible, challenging framework for evaluating and advancing LLMs in multi-function, compositional code generation scenarios, highlighting the current gap in chaining and code reuse abilities [2412.21199].

Source: https://www.emergentmind.com/topics/mbpp-pro