Mostly Basic Programming Problems (MBPP)

• MBPP is a dataset of 974 entry-level Python programming challenges that assess basic function-level code generation using natural language prompts.
• The dataset emphasizes numeric, list, and string manipulations, serving as a benchmark for evaluating neural and human-in-the-loop programming methods.
• Research with MBPP has driven advances in error analysis, model scaling, and modular prompting techniques to enhance code synthesis performance.

Mostly Basic Programming Problems (MBPP) are a foundational dataset and benchmarking resource in the evaluation of program synthesis, particularly function-level code generation via natural language prompts. Originally formulated to reflect challenges solvable by entry-level programmers, MBPP has played a pivotal role in shaping the evolution of neural code generation methods, human-in-the-loop programming workflows, and the critical assessment of model limitations.

1. Definition and Construction

The MBPP dataset consists of 974 Python programming tasks, each designed to be solvable by entry-level programmers through short, self-contained functions (2108.07732). Each problem includes:

• A natural language description specifying the desired functionality.
• A function signature and canonical solution implementation.
• Three assert-based test cases, which serve as the gold standard for semantic correctness.

The construction aimed to encompass "mostly basic" programming problems: tasks involving numeric manipulation, list and string processing, and simple control flow, thereby ensuring accessibility and relevance to introductory programming education. The problems were crowdsourced and curated, and ambiguous problem statements were later revised to ensure clarity and consistency.

2. Characteristics and Scope

MBPP tasks primarily emphasize basic programming concepts:

• Approximately 58% are mathematically oriented (e.g., arithmetic, conversion).
• 43% involve list operations (e.g., filtering, mapping, aggregation).
• 19% require string manipulation.
• A small minority involve other sequence and data structure tasks (2108.07732).

Problems are intentionally "mostly basic": they leverage standard library functions where possible and avoid advanced data structures or object-oriented paradigms. Task descriptions in the original set average 15.7 words, with concise associated test suites (three test cases per problem).

3. Benchmarking and Evaluation Protocols

MBPP has become a central benchmark for evaluating both enumerative search-based and neural program synthesis methods. The primary evaluation metric is Pass@1: the fraction of problems for which a model produces a function passing all provided test cases on the first attempt. If multiple samples are generated per prompt, the Pass@k metric is also reported.

In practice, a canonical prompt for an MBPP task includes the description, function signature, and several assert statements defining the expected I/O behavior. Models are evaluated by generating candidate code solutions and executing them against these asserts. An "edited" version of MBPP was released to address minor inconsistencies and clarify requirements (2108.07732).

MBPP's design has encouraged rigorous error analysis—models are evaluated not just for pass rates, but also for the character of their failures, including syntax errors, runtime errors, or failing semantic assertions.

4. Advances in Model Performance and Methodology

LLMs have delivered significant advances in MBPP benchmark performance. Key empirical findings include:

• Synthesis accuracy scales approximately linearly with the logarithm of model size:

$$\text{Accuracy} \approx a \cdot \log(\text{Number of Parameters}) + b$$

• The largest pre-2022 models (137B parameters) achieved ~59.6% Pass@1 with few-shot prompting, rising to ~70% after targeted fine-tuning (2108.07732).
• Human-in-the-loop corrections (natural language feedback across several dialog turns) reduced error rates by half compared to initial predictions.
• Planning-driven workflows, such as the two-phase LPW method, decomposing problems into natural language plans with verification against visible tests, push Pass@1 as high as 84.8% (using GPT-4o) (2411.14503).

Advanced prompting techniques, notably Modular Prompting (MoT), which decompose problem-solving into hierarchical, modular reasoning steps, yield further improvements in both accuracy and interpretability. MoT achieved a Pass@1 of 73.9% on MBPP using GPT-4o-mini, outperforming baseline methods like standard Chain-of-Thought (CoT) (2503.12483).

5. Error Analysis, Model Limitations, and Language Migration

Error analysis across several studies reveals persistent limitations:

• Small models are prone to syntax and type errors, while larger models mainly fail on semantic assertions where generated code does not fully capture intent (2108.07732).
• Complex problems requiring multi-step reasoning (e.g., subsequence or multi-constraint tasks) pose consistent challenges.
• Overfitting to test assertions remains rare but notable (e.g., returning hardcoded outputs matching specific asserts).
• Language confusion and strategic migration: models, especially when prompted for non-Python languages, systematically default to Python if uncertain, prioritizing syntactic validity over target language fidelity—an effect that persists even for basic MBPP problems (2503.13620).

Table: Model Pass Rates and Confusion Metrics on MBPP

Model	Pass@1 (%)	LCPR (%)	CPPR (%)
GPT-4o	84.8	~90	99+
CodeLlama-34B	~70	~50	99+
Mistral-7B	~65	~90	99+

LCPR: Language Confusion Pass Rate. CPPR: Code Parsing Pass Rate.

6. Dataset Limitations, Contamination, and the Evolution of Benchmarks

Recent analyses identify several drawbacks with MBPP:

• Data contamination: Approximately 65.4% of MBPP test instances have been obtained from open-access websites, suggesting that powerful models may "cheat" via memorization rather than reasoning (2405.11430).
• Quality and granularity: MBPP’s task descriptions are concise, and its problem spectrum is skewed (77% mathematical or list operations), limiting its ability to discriminate among advanced models.
• Low challenge ceiling: Many models have saturated MBPP, necessitating more challenging benchmarks.

These limitations have motivated the development of harder datasets, notably MHPP (Mostly Hard Python Problems), which exhibits tenfold longer problem descriptions (150.2 words versus 15.7), larger test suites (average 13.5 vs. 3.0), and a much broader and deeper set of coding challenges (2405.11430). MHPP highlights model weaknesses invisible in MBPP, such as difficulty with distraction, multifaceted reasoning, and nuanced "codesense" skills.

7. Dataset Augmentation and Future Directions

To address diversity and realism, automated dataset augmentation has become an active area:

• Programming Problem Merging (PPM) produces new programming challenges by semantically recombining MBPP tasks using controlled metamorphic transformations (PPM-V, PPM-T), dramatically increasing prompt and solution diversity (2401.15545).
• Such methodologies expose new model weaknesses (e.g., 85–90% drops in Pass@1 on merged problems) and reduce data leakage risks compared to surface-level perturbations or manual editing.

Further, retrieval-augmented generation approaches leveraging programming knowledge graphs (PKG) now support fine-grained code retrieval and context selection, with measurable improvements in MBPP Pass@1 (up to 20% over baselines for DeepSeek-Coder-7B) (2410.18251).

Visible test-driven iterative workflows, modular reasoning structures, and in-execution self-debugging constitute additional current directions aimed at enhancing both the reliability and human-alignment of MBPP problem solving (2411.14503, 2501.12793, 2503.12483).

Conclusion

MBPP serves as a crucial testbed for code synthesis research, establishing a baseline for measuring the abilities of program synthesis systems to generate correct, maintainable solutions to well-specified programming problems. Its limitations relating to data contamination, challenge granularity, and language fidelity have spurred the design of harder, more diverse benchmarks, novel augmentation strategies, and richer evaluation methodologies. Ongoing research continues to leverage MBPP for method development, while using its observed weaknesses as a springboard for the creation of more robust and discriminative assessments of code generation competence.