MP-Bench: Multi-Domain Benchmark Overview

• MP-Bench is a context-dependent benchmark framework spanning MPI microbenchmarking, probabilistic sample assessment, MPC repair, and multi-path mobile-agent evaluation.
• It emphasizes experimental reproducibility through rigorous methods such as repeated independent runs, drift-aware synchronization, and statistical inference.
• The framework highlights the limitations of single scalar metrics by advocating for process-aware, reference-based evaluations tailored to each domain.

“MP-Bench” is not introduced in the cited arXiv literature as a single canonical benchmark title. Instead, the term is best treated as context-dependent. In the narrowest sense, it appears in the MPI benchmarking literature as part of the ecosystem of MPI microbenchmark suites; in adjacent work, closely related benchmark names include MCBench for Monte Carlo sample-quality assessment, MPC-Patch-Bench for repository-level repair in secure multi-party computation software, and Mobile-Bench and Mobile-Bench-v2 for mobile-agent evaluation. This suggests that “MP-Bench” functions less as a unique artifact than as a shorthand whose meaning depends on whether the surrounding discourse concerns MPI benchmarking, multi-party computation, multi-path mobile-agent evaluation, or sample-based probabilistic inference (Hunold et al., 2015, Ding et al., 6 Jan 2025, Zhang et al., 9 Jun 2026, Deng et al., 2024, Xu et al., 17 May 2025).

1. Terminological status and principal referents

A useful way to situate the term is to distinguish the benchmark lines that the cited literature actually defines.

Benchmark line	Domain	Core evaluand
MCBench	Monte Carlo sampling	Distributional fidelity of samples
MPC-Patch-Bench	MPC software repair	Functional correctness plus cryptographic safety
Mobile-Bench / Mobile-Bench-v2	Mobile agents	Sequential execution quality, multi-app use, multi-path behavior
MP-Bench in MPI usage	MPI microbenchmarking	Performance measurement methodology

In this disambiguated view, the most literal occurrence of MP-Bench is in the MPI paper, which treats it as one of several established benchmark suites and argues that the main unresolved problem is not the existence of benchmark kernels, but the reproducibility and statistical validity of benchmark results. By contrast, Mobile-Bench-v2 explicitly states that it is not literally named MP-Bench, while also noting that it can be regarded as “MP” in the sense of a multi-path benchmark. MCBench is described as relevant to an “MP-Bench-style effort” for probabilistic inference, and MPC-Patch-Bench provides the most direct expansion of “MP” as multi-party computation (Hunold et al., 2015, Xu et al., 17 May 2025, Ding et al., 6 Jan 2025, Zhang et al., 9 Jun 2026).

2. MP-Bench in the MPI benchmarking literature

In MPI usage, MP-Bench belongs to the class of benchmark suites used to assess the performance of MPI implementations across message sizes, collective operations, and supercomputer architectures. The methodological intervention in “MPI Benchmarking Revisited: Experimental Design and Reproducibility” is that many MPI benchmark results are “neither reproducible nor statistically sound,” even when the underlying benchmark kernels are familiar. The paper focuses on blocking collective MPI operations and treats suites such as MP-Bench, Intel MPI Benchmarks, SKaMPI, OSU Micro-Benchmarks, NBCBench, MPIBench, and MPIBlib as representative of this ecosystem (Hunold et al., 2015).

The central claim is that benchmark outcomes depend strongly on experimental factors that are frequently underreported or uncontrolled. The paper identifies, among others, synchronization method, clock offset and drift, the choice of mpirun as an experimental replication unit, system noise, number of repetitions, cache state, process pinning, compiler and compiler flags, DVFS, window size, and timing source. It shows that different invocations of the same benchmark can yield materially different results, that MPI_Barrier does not ensure synchronous barrier exit, and that offset-only clock synchronization degrades over time because clock drift accumulates (Hunold et al., 2015).

The proposed response is an explicitly statistical and reproducible protocol. The paper recommends repeated independent mpirun launches, many measurements per launch, randomization of case order within each launch, outlier removal with Tukey’s filter, and cross-implementation comparison by the Wilcoxon–Mann–Whitney test at $\alpha = 0.05$. It also proposes HCA (Hunold–Carpen-Amarie), a drift-aware clock-synchronization method that combines a linear drift model with hierarchical synchronization. In this sense, MP-Bench is best understood not merely as a set of timed MPI calls, but as a benchmarking practice whose validity depends on synchronization, replication, and inference methodology (Hunold et al., 2015).

3. MCBench as a probabilistic-inference analogue

In probabilistic inference, the closest explicit analogue to an “MP-Bench-style” benchmark is MCBench, a domain-neutral benchmark suite for Monte Carlo sample quality assessment. Its objective is not task accuracy or runtime alone, but the distributional fidelity of the samples produced by a sampler when compared against trusted IID reference samples from the same target distribution. The benchmark is therefore centered on the question: given a set of samples from an external Monte Carlo method, how well do those samples reproduce the intended target distribution under finite-sample and correlated-sample conditions? (Ding et al., 6 Jan 2025)

MCBench requires targets to be IID-sampleable and uses internally generated IID draws as the gold-standard reference. External samplers can then be evaluated by repeated batched comparisons under both basic statistical metrics and advanced two-sample discrepancies. The paper explicitly mentions mean, variance, and chi-square test statistics, and gives special emphasis to the sliced Wasserstein distance and maximum mean discrepancy. The benchmark’s target suite spans Standard Normal 1D, Standard Normal $k$D in uncorrelated and correlated variants, Mixture Normal $k$D with strong correlation, Cauchy 1D, and the eight schools example from posteriorDB. The stated purpose of this design is to probe dimensionality, correlation structure, multimodality, heavy tails, and practical relevance (Ding et al., 6 Jan 2025).

A distinctive feature is the way metrics are transformed into benchmark statistics. MCBench does not rely on a single one-off discrepancy value. Instead, it builds empirical metric distributions from repeated IID vs IID comparisons and contrasts them with repeated IID vs user-sample comparisons. User samples are partitioned into batches with an effective sample size, using by default the weight-based formula $\mathrm{ESS} \approx (\sum_i w_i)^2 / \sum_i w_i^2$, although the paper notes that ESS can be implemented individually depending on the sampling method. The final outputs include per-metric empirical means and standard deviations, normalized comparison plots with $1\sigma$, $2\sigma$, and $3\sigma$ regions, histograms via plot_teststatistic, and JSON serialization of test statistics (Ding et al., 6 Jan 2025).

The implementation is a Julia package organized around the abstractions test case, sampler, and metric. It supports built-in and custom targets, direct Julia-side samplers, and external imports through FileBasedSampler, with examples for Stan, PyMC, and BAT.jl. The case studies show both the benchmark’s sensitivity and its dependence on calibration: on a 3D uncorrelated Gaussian, Metropolis–Hastings appeared broadly consistent with the IID baseline but showed a variance mismatch attributed to overestimation of the effective sample size; on a correlated multimodal mixture, the same method failed clearly, with incorrect relative mode strengths and strong deviations from the IID baseline (Ding et al., 6 Jan 2025).

4. MPC-Patch-Bench and the multi-party computation interpretation

If “MP” is read as multi-party, the most direct benchmark in the cited literature is MPC-Patch-Bench, presented as the first repository-level benchmark for LLM code repair on real-world MPC software. Its defining claim is that ordinary repository-level repair benchmarks such as SWE-bench are structurally inadequate for MPC. The paper gives three reasons: MPC repositories are dominated by generic infrastructure rather than cryptographic logic; high-value MPC fixes often lack standardized tests; and ordinary fail-to-pass evaluation is insufficient because a patch may pass tests while still violating secrecy, data-obliviousness, type discipline, or numerical fidelity (Zhang et al., 9 Jun 2026).

The benchmark contains 205 fully verified, executable task instances curated from CrypTen, tf-encrypted, MP-SPDZ, SecretFlow, and PySyft, starting from a raw corpus of 7,305 merged pull requests. Its Data Curation Framework uses a Domain-Specific Curation Agent with three filtering layers—Primitive Check, Protocol Logic, and Domain Specificity—followed by a Human-AI Completion Engine that synthesizes missing problem statements, Fail-to-Pass (F2P) and Pass-to-Pass (P2P) tests, an execution harness, and a verified gold patch. Confidence-based routing uses $\tau_{high} = 0.9$ and $\tau_{med} = 0.75$, and the reported curation yield is $7{,}305 \rightarrow 1{,}175 \rightarrow 205$, a 4.9× expansion over strict SWE-bench-style extraction on the same corpus, which would have yielded 42 instances (Zhang et al., 9 Jun 2026).

The benchmark’s second core component is the MPC Verifier, which applies dynamic differential testing against plaintext oracles and domain-specific SAST after ordinary functional resolution. Dynamic checking includes tolerance validation with boundary values such as $k$0, $k$1, and $k$2, together with numerical-fidelity checks using tight bounds such as $k$3. The static stream flags four classes of anti-patterns: unsafe reveals, insecure arithmetic, illegal type casting, and context-aware crypto-rule violations. A patch is functionally resolved iff all F2P tests pass and all P2P tests continue to pass, but verified resolution requires surviving both dynamic and static verification as well (Zhang et al., 9 Jun 2026).

The reported results show why this stricter correctness contract matters. The strongest functional result is 22.9%, while the best verified resolution rate is 17.1%; the verifier rejects up to 40% of functionally passing patches for cryptographic or numerical-fidelity violations. The verifier can also reorder model rankings, which underscores that in MPC repair, test passing is not equivalent to acceptable repair (Zhang et al., 9 Jun 2026).

5. Mobile-Bench, Mobile-Bench-v2, and the multi-path reading of “MP”

In the mobile-agent literature, Mobile-Bench defines a benchmark for LLM-based mobile agents that act on an Android phone through a hybrid UI + API interface. It contains 832 data entries, organized into SAST, SAMT, and MAMT, and includes 103 collected APIs across 29 applications. The benchmark is motivated by three deficiencies in earlier mobile-agent evaluation: UI-only interaction is inefficient, single-app instructions are too narrow to test planning, and final-outcome metrics are inadequate for sequential action evaluation. Its principal methodological contribution is CheckPoint, a process-sensitive metric that scores whether the agent reaches essential intermediate states at package, key-phrase, and API levels (Deng et al., 2024).

Mobile-Bench executes in an Android 14.0 emulator with Appium UiAutomator2 Driver. UI XML is converted to HTML, the action space includes Click, Scroll, Input, and API Call, and evaluation reports PassRate, Average Steps, and CheckPoint. The empirical results show a sharp difficulty gradient from SAST to MAMT, with GPT-4 performing best overall and API access substantially improving both efficiency and procedural correctness. The benchmark’s main claim is that endpoint success alone overestimates capability because models may stop early or misjudge completion (Deng et al., 2024).

Mobile-Bench-v2 extends this line to VLM-based mobile agents and explicitly introduces offline multi-path evaluation. Built on Mobile3M, a graph-structured corpus with 20M+ user interactions, 3M screenshots, and 49 widely used Chinese apps across 15 categories, it uses the GIAS pipeline to generate slot-based instructions from mobile UI action sequences. The common split contains 12,854 instructions, supplemented by a noisy split of 100 instructions, a contaminated AITZ-Noise split with ads inserted into 2,504 trajectories, and an ambiguous-instruction split of 100 instructions with preset Q&A for proactive interaction. Because evaluation cost is high, the paper reports zero-shot experiments on Random-800, an 800-instruction subset of the common split (Xu et al., 17 May 2025).

The paper states explicitly that there is no benchmark in this paper literally named MP-Bench, but also argues that if “MP-Bench” is understood as a multi-path benchmark, then Mobile-Bench-v2 qualifies in spirit. Its metrics are Success Rate (SR), Step Efficiency (SE), Step.Acc, and TYPE, and its main findings are that multi-path evaluation reveals differences obscured by single-path imitation, that current agents are extremely brittle under real noisy interfaces, and that structured clarification under ambiguous instructions can improve downstream action accuracy (Xu et al., 17 May 2025).

6. Shared design principles, misconceptions, and limits

Across these benchmark lines, a common methodological pattern is the rejection of a single scalar endpoint as an adequate notion of correctness. MCBench argues that runtime or task-level success alone does not establish whether a sampler reproduces the target distribution; MPC-Patch-Bench argues that F2P/P2P success alone does not establish cryptographic safety; Mobile-Bench argues that PassRate alone does not capture the quality of sequential execution; and the MPI study argues that a single benchmark run summarized by a mean or minimum does not establish a reproducible performance claim (Ding et al., 6 Jan 2025, Zhang et al., 9 Jun 2026, Deng et al., 2024, Hunold et al., 2015).

This suggests that the benchmark philosophy associated with “MP-Bench” in contemporary usage is increasingly reference-based, process-aware, and domain-specific. The reference may be IID samples, a plaintext oracle, key nodes in a graph-structured UI corpus, or repeated independent mpirun launches. The evaluation layer is then calibrated against that reference rather than inferred from a final output alone (Ding et al., 6 Jan 2025, Zhang et al., 9 Jun 2026, Xu et al., 17 May 2025, Hunold et al., 2015).

The same literature also makes clear that each benchmark family has bounded scope. MCBench is largely confined to sample quality and requires IID-reference-available targets; MPC-Patch-Bench currently covers five widely used MPC frameworks and does not provide a formal soundness or completeness proof for its verifier; Mobile-Bench-v2 states that offline multi-path evaluation is not fully equivalent to online execution and cannot exhaustively cover the range of text inputs; and the MPI methodology paper is empirically centered on blocking collective MPI operations rather than all MPI function classes (Ding et al., 6 Jan 2025, Zhang et al., 9 Jun 2026, Xu et al., 17 May 2025, Hunold et al., 2015).

Accordingly, “MP-Bench” is best treated as a disambiguation problem rather than a single settled benchmark object. In MPI it denotes a microbenchmark-suite tradition whose central issue is reproducible experimental design; in MPC it points to security-aware repository repair; in mobile-agent research it aligns most closely with multi-path evaluation; and in probabilistic inference it is exemplified by reference-based assessment of sample quality.