---
title: 'MP-Bench: Multi-Domain Benchmark Overview'
url: https://www.emergentmind.com/topics/mp-bench
type: topic
---

# MP-Bench: Multi-Domain Benchmark Overview

“MP-Bench” is not introduced in the cited arXiv literature as a single canonical benchmark title. Instead, the term is best treated as context-dependent. In the narrowest sense, it appears in the MPI benchmarking literature as part of the ecosystem of MPI microbenchmark suites; in adjacent work, closely related benchmark names include **MCBench** for Monte Carlo sample-quality assessment, **MPC-Patch-Bench** for repository-level repair in secure multi-party computation software, and **Mobile-Bench** and **Mobile-Bench-v2** for mobile-agent evaluation. This suggests that “MP-Bench” functions less as a unique artifact than as a shorthand whose meaning depends on whether the surrounding discourse concerns **MPI benchmarking**, **multi-party computation**, **multi-path mobile-agent evaluation**, or **sample-based probabilistic inference** [1505.07734], [2501.03138], [2606.11416], [2407.00993], [2505.11891].

## 1. Terminological status and principal referents

A useful way to situate the term is to distinguish the benchmark lines that the cited literature actually defines.

| Benchmark line | Domain | Core evaluand |
|---|---|---|
| MCBench | Monte Carlo sampling | Distributional fidelity of samples |
| MPC-Patch-Bench | MPC software repair | Functional correctness plus cryptographic safety |
| Mobile-Bench / Mobile-Bench-v2 | Mobile agents | Sequential execution quality, multi-app use, multi-path behavior |
| MP-Bench in MPI usage | MPI microbenchmarking | Performance measurement methodology |

In this disambiguated view, the most literal occurrence of **MP-Bench** is in the MPI paper, which treats it as one of several established benchmark suites and argues that the main unresolved problem is not the existence of benchmark kernels, but the reproducibility and statistical validity of benchmark results. By contrast, **Mobile-Bench-v2** explicitly states that it is *not* literally named MP-Bench, while also noting that it can be regarded as “MP” in the sense of a **multi-path benchmark**. **MCBench** is described as relevant to an “MP-Bench-style effort” for probabilistic inference, and **MPC-Patch-Bench** provides the most direct expansion of “MP” as **multi-party computation** [1505.07734], [2505.11891], [2501.03138], [2606.11416].

## 2. MP-Bench in the MPI benchmarking literature

In MPI usage, MP-Bench belongs to the class of benchmark suites used to assess the performance of MPI implementations across message sizes, collective operations, and supercomputer architectures. The methodological intervention in “MPI Benchmarking Revisited: Experimental Design and Reproducibility” is that many MPI benchmark results are “neither reproducible nor statistically sound,” even when the underlying benchmark kernels are familiar. The paper focuses on **blocking collective MPI operations** and treats suites such as **MP-Bench**, Intel MPI Benchmarks, SKaMPI, OSU Micro-Benchmarks, NBCBench, MPIBench, and MPIBlib as representative of this ecosystem [1505.07734].

The central claim is that benchmark outcomes depend strongly on experimental factors that are frequently underreported or uncontrolled. The paper identifies, among others, **synchronization method**, **clock offset and drift**, the choice of **`mpirun`** as an experimental replication unit, **system noise**, **number of repetitions**, **cache state**, **process pinning**, **compiler and compiler flags**, **DVFS**, **window size**, and **timing source**. It shows that different invocations of the same benchmark can yield materially different results, that `MPI_Barrier` does not ensure synchronous barrier exit, and that offset-only clock synchronization degrades over time because clock drift accumulates [1505.07734].

The proposed response is an explicitly statistical and reproducible protocol. The paper recommends repeated independent **`mpirun` launches**, many measurements per launch, randomization of case order within each launch, outlier removal with **Tukey’s filter**, and cross-implementation comparison by the **Wilcoxon–Mann–Whitney test** at \(\alpha = 0.05\). It also proposes **HCA** (Hunold–Carpen-Amarie), a drift-aware clock-synchronization method that combines a linear drift model with hierarchical synchronization. In this sense, MP-Bench is best understood not merely as a set of timed MPI calls, but as a benchmarking practice whose validity depends on synchronization, replication, and inference methodology [1505.07734].

## 3. MCBench as a probabilistic-inference analogue

In probabilistic inference, the closest explicit analogue to an “MP-Bench-style” benchmark is **MCBench**, a **domain-neutral benchmark suite for Monte Carlo sample quality assessment**. Its objective is not task accuracy or runtime alone, but the **distributional fidelity of the samples produced by a sampler** when compared against trusted IID reference samples from the same target distribution. The benchmark is therefore centered on the question: given a set of samples from an external Monte Carlo method, how well do those samples reproduce the intended target distribution under finite-sample and correlated-sample conditions? [2501.03138]

MCBench requires targets to be **IID-sampleable** and uses internally generated IID draws as the gold-standard reference. External samplers can then be evaluated by repeated batched comparisons under both **basic statistical metrics** and **advanced two-sample discrepancies**. The paper explicitly mentions **mean**, **variance**, and **chi-square test statistics**, and gives special emphasis to the **sliced Wasserstein distance** and **maximum mean discrepancy**. The benchmark’s target suite spans **Standard Normal 1D**, **Standard Normal \(k\)D** in uncorrelated and correlated variants, **Mixture Normal \(k\)D** with strong correlation, **Cauchy 1D**, and the **eight schools example** from posteriorDB. The stated purpose of this design is to probe dimensionality, correlation structure, multimodality, heavy tails, and practical relevance [2501.03138].

A distinctive feature is the way metrics are transformed into benchmark statistics. MCBench does not rely on a single one-off discrepancy value. Instead, it builds empirical metric distributions from repeated **IID vs IID** comparisons and contrasts them with repeated **IID vs user-sample** comparisons. User samples are partitioned into batches with an **effective sample size**, using by default the weight-based formula \(\mathrm{ESS} \approx (\sum_i w_i)^2 / \sum_i w_i^2\), although the paper notes that ESS can be implemented individually depending on the sampling method. The final outputs include per-metric empirical means and standard deviations, normalized comparison plots with \(1\sigma\), \(2\sigma\), and \(3\sigma\) regions, histograms via `plot_teststatistic`, and JSON serialization of test statistics [2501.03138].

The implementation is a **Julia package** organized around the abstractions **test case**, **sampler**, and **metric**. It supports built-in and custom targets, direct Julia-side samplers, and external imports through `FileBasedSampler`, with examples for **Stan**, **PyMC**, and **BAT.jl**. The case studies show both the benchmark’s sensitivity and its dependence on calibration: on a **3D uncorrelated Gaussian**, Metropolis–Hastings appeared broadly consistent with the IID baseline but showed a variance mismatch attributed to **overestimation of the effective sample size**; on a **correlated multimodal mixture**, the same method failed clearly, with incorrect relative mode strengths and strong deviations from the IID baseline [2501.03138].

## 4. MPC-Patch-Bench and the multi-party computation interpretation

If “MP” is read as **multi-party**, the most direct benchmark in the cited literature is **MPC-Patch-Bench**, presented as the **first repository-level benchmark for LLM code repair on real-world MPC software**. Its defining claim is that ordinary repository-level repair benchmarks such as SWE-bench are structurally inadequate for MPC. The paper gives three reasons: MPC repositories are dominated by generic infrastructure rather than cryptographic logic; high-value MPC fixes often lack standardized tests; and ordinary fail-to-pass evaluation is insufficient because a patch may pass tests while still violating secrecy, data-obliviousness, type discipline, or numerical fidelity [2606.11416].

The benchmark contains **205 fully verified, executable task instances** curated from **CrypTen**, **tf-encrypted**, **MP-SPDZ**, **SecretFlow**, and **PySyft**, starting from a raw corpus of **7,305 merged pull requests**. Its **Data Curation Framework** uses a **Domain-Specific Curation Agent** with three filtering layers—**Primitive Check**, **Protocol Logic**, and **Domain Specificity**—followed by a **Human-AI Completion Engine** that synthesizes missing problem statements, **Fail-to-Pass (F2P)** and **Pass-to-Pass (P2P)** tests, an execution harness, and a verified gold patch. Confidence-based routing uses \(\tau_{high} = 0.9\) and \(\tau_{med} = 0.75\), and the reported curation yield is \(7{,}305 \rightarrow 1{,}175 \rightarrow 205\), a **4.9× expansion** over strict SWE-bench-style extraction on the same corpus, which would have yielded **42** instances [2606.11416].

The benchmark’s second core component is the **MPC Verifier**, which applies **dynamic differential testing against plaintext oracles** and **domain-specific SAST** after ordinary functional resolution. Dynamic checking includes tolerance validation with boundary values such as \(0\), \(-1\), and \(\text{MAX\_INT}\), together with numerical-fidelity checks using tight bounds such as \(\text{atol} = 0.005\). The static stream flags four classes of anti-patterns: **unsafe reveals**, **insecure arithmetic**, **illegal type casting**, and **context-aware crypto-rule violations**. A patch is functionally resolved iff all F2P tests pass and all P2P tests continue to pass, but verified resolution requires surviving both dynamic and static verification as well [2606.11416].

The reported results show why this stricter correctness contract matters. The strongest functional result is **22.9%**, while the best **verified resolution rate** is **17.1%**; the verifier rejects **up to 40%** of functionally passing patches for cryptographic or numerical-fidelity violations. The verifier can also reorder model rankings, which underscores that in MPC repair, test passing is not equivalent to acceptable repair [2606.11416].

## 5. Mobile-Bench, Mobile-Bench-v2, and the multi-path reading of “MP”

In the mobile-agent literature, **Mobile-Bench** defines a benchmark for **LLM-based mobile agents** that act on an Android phone through a hybrid **UI + API** interface. It contains **832 data entries**, organized into **SAST**, **SAMT**, and **MAMT**, and includes **103 collected APIs** across **29 applications**. The benchmark is motivated by three deficiencies in earlier mobile-agent evaluation: UI-only interaction is inefficient, single-app instructions are too narrow to test planning, and final-outcome metrics are inadequate for sequential action evaluation. Its principal methodological contribution is **CheckPoint**, a process-sensitive metric that scores whether the agent reaches essential intermediate states at package, key-phrase, and API levels [2407.00993].

Mobile-Bench executes in an **Android 14.0 emulator** with **Appium UiAutomator2 Driver**. UI XML is converted to HTML, the action space includes **Click**, **Scroll**, **Input**, and **API Call**, and evaluation reports **PassRate**, **Average Steps**, and **CheckPoint**. The empirical results show a sharp difficulty gradient from **SAST** to **MAMT**, with **GPT-4** performing best overall and API access substantially improving both efficiency and procedural correctness. The benchmark’s main claim is that endpoint success alone overestimates capability because models may stop early or misjudge completion [2407.00993].

**Mobile-Bench-v2** extends this line to **VLM-based mobile agents** and explicitly introduces **offline multi-path evaluation**. Built on **Mobile3M**, a graph-structured corpus with **20M+ user interactions**, **3M screenshots**, and **49 widely used Chinese apps** across **15 categories**, it uses the **GIAS** pipeline to generate slot-based instructions from mobile UI action sequences. The common split contains **12,854** instructions, supplemented by a **noisy split** of **100** instructions, a contaminated **AITZ-Noise** split with ads inserted into **2,504 trajectories**, and an **ambiguous-instruction split** of **100** instructions with preset Q\&A for proactive interaction. Because evaluation cost is high, the paper reports zero-shot experiments on **Random-800**, an 800-instruction subset of the common split [2505.11891].

The paper states explicitly that there is **no benchmark in this paper literally named MP-Bench**, but also argues that if “MP-Bench” is understood as a **multi-path benchmark**, then Mobile-Bench-v2 qualifies in spirit. Its metrics are **Success Rate (SR)**, **Step Efficiency (SE)**, **Step.Acc**, and **TYPE**, and its main findings are that multi-path evaluation reveals differences obscured by single-path imitation, that current agents are extremely brittle under real noisy interfaces, and that structured clarification under ambiguous instructions can improve downstream action accuracy [2505.11891].

## 6. Shared design principles, misconceptions, and limits

Across these benchmark lines, a common methodological pattern is the rejection of a single scalar endpoint as an adequate notion of correctness. **MCBench** argues that runtime or task-level success alone does not establish whether a sampler reproduces the target distribution; **MPC-Patch-Bench** argues that F2P/P2P success alone does not establish cryptographic safety; **Mobile-Bench** argues that PassRate alone does not capture the quality of sequential execution; and the MPI study argues that a single benchmark run summarized by a mean or minimum does not establish a reproducible performance claim [2501.03138], [2606.11416], [2407.00993], [1505.07734].

This suggests that the benchmark philosophy associated with “MP-Bench” in contemporary usage is increasingly **reference-based**, **process-aware**, and **domain-specific**. The reference may be IID samples, a plaintext oracle, key nodes in a graph-structured UI corpus, or repeated independent `mpirun` launches. The evaluation layer is then calibrated against that reference rather than inferred from a final output alone [2501.03138], [2606.11416], [2505.11891], [1505.07734].

The same literature also makes clear that each benchmark family has bounded scope. MCBench is largely confined to **sample quality** and requires **IID-reference-available targets**; MPC-Patch-Bench currently covers **five widely used MPC frameworks** and does not provide a formal soundness or completeness proof for its verifier; Mobile-Bench-v2 states that offline multi-path evaluation is **not fully equivalent to online execution** and cannot exhaustively cover the range of text inputs; and the MPI methodology paper is empirically centered on **blocking collective MPI operations** rather than all MPI function classes [2501.03138], [2606.11416], [2505.11891], [1505.07734].

Accordingly, “MP-Bench” is best treated as a disambiguation problem rather than a single settled benchmark object. In MPI it denotes a microbenchmark-suite tradition whose central issue is reproducible experimental design; in MPC it points to security-aware repository repair; in mobile-agent research it aligns most closely with **multi-path** evaluation; and in probabilistic inference it is exemplified by reference-based assessment of **sample quality**.

Source: https://www.emergentmind.com/topics/mp-bench