MPBench: Multiparty Optimization & Evaluation

• MPBench is a collection of benchmark suites that rigorously test multiparty evaluation tasks in optimization, multimodal reasoning, AI-generated image detection, and mobile inference.
• It features analytic test problems with known Pareto sets and realistic UAV path planning scenarios under diverse operational constraints.
• It provides unified metrics, such as MPIGD and MPHV, to compare algorithm performance across 17 standardized test cases in competitive research.

MPBench is a designation held by several distinct benchmark suites across different research domains, each serving as a rigorous, open-source evaluation standard for models or algorithms at the forefront of their respective fields. The following survey catalogs key MPBench instances: (1) as a comprehensive testbed for process-level reward models in multimodal reasoning, (2) as a model perception benchmark for detecting AI-generated images, (3) as the official benchmark for multiparty multiobjective optimization at the CEC 2024 competition, and (4) as the MLPerf Mobile Inference Benchmark (commonly abbreviated MPBench) for cross-platform device ML evaluation. Each version is independent, and the label "MPBench" must be contextualized by its research community.

1. Multiparty Multiobjective Optimization: MPBench for CEC 2024

MPBench for CEC 2024 formalizes the multiparty multiobjective optimization problem (MPMOP) paradigm, in which $M$ distinct decision makers (DMs) each possess vector-valued objective functions, and a solution is considered multiparty-optimal if no feasible candidate is weakly better for all DMs and strictly better for at least one. Formally, the MPMOP is:

$$\min\;E(\mathbf x)\;=\;\bigl[F_1(\mathbf x),\,F_2(\mathbf x),\dots,F_M(\mathbf x)\bigr],$$

with $$F_i(\mathbf x) = (f_{i1}(\mathbf x), \dots, f_{im_i}(\mathbf x))$$ representing the objective suite for DM $i$.

Benchmark Structure

MPBench divides its suite into two major classes:

• Part I: Analytic Test Problems with Known Pareto Sets
  • 11 MPMOPs, derived by combining time-indexed dynamic MOP functions (BF1–BF6 from CEC’2018, via Liu et al. 2020). Each problem incorporates multiple DMs (either pairs or triplets), each assigned objective surfaces at different time-slices ($t$). All share a nonempty analytical Pareto-optimal set.
  • Example construction:

  $$E_1(\mathbf x) = \bigl[ F_{11}(\mathbf x), F_{12}(\mathbf x) \bigr],$$

  where $$F_{11}(\mathbf x) = [ f_{11}(\mathbf x, 1), f_{12}(\mathbf x, 1) ]$$ and $$F_{12}(\mathbf x) = [ f_{11}(\mathbf x, 2), f_{12}(\mathbf x, 2) ]$$. - Problem dimensions: $n \in \{10, 30, 50\}$.

• Part II: Realistic Path Planning Scenarios

  • Six biparty UAV path planning (BP-UAVPP) problems, in which DMs represent efficiency (minimizing trajectory cost, fuel, height change, and distance to hover-points) and safety (minimizing fatality risk, property risk, noise), over discrete trajectory waypoints under combined terrain and vehicle constraints. Objective formulas, such as $f_{\text{fuel}}$ and $f_{\text{fatal}}$, are given precisely in the benchmark description.
  • Six scenarios ($C_1$–$C_6$) vary in assignment and inclusion of objectives for each party.

2. Evaluation Metrics and Competition Protocol

• Part I evaluation (Pareto known): Multiparty Inverted Generational Distance (MPIGD):

$$\operatorname{MPIGD}(P^{MP},P) = \frac{1}{|P^{MP}|}\sum_{v \in P^{MP}} d(v, P),$$

with

$$d(v, P) = \min_{s \in P} \sum_{j=1}^M \sqrt{ \sum_{\ell=1}^{m_j} (v_{j\ell} - s_{j\ell})^2 }.$$

• Part II evaluation (Pareto unknown): Multiparty Hypervolume (MPHV), summed or averaged across parties:

$\mathrm{MPHV} = \frac{1}{M} \sum_{i=1}^M HV_i,$

where $HV_i$ is the individual hypervolume achieved by DM $i$’s solution set.

• Ranking: The final benchmark score is the average algorithm rank across all problems (all 17 test cases).
• Competition protocol: Dimension, maximum function evaluations ($\mathrm{FE}_{\max}$), and number of independent runs (30) are fixed; all other details (algorithm choice, population sizes, parameter settings) are at the discretion of contestants. No baseline or comparative results are supplied in the benchmark paper; these will be established post-competition by aggregating MPIGD/MPHV metrics and rankings (Luo et al., 2024).

3. Reference Algorithms and Literature Context

While MPBench itself does not prescribe baselines or report empirical results, it catalogs established approaches such as:

• OptMPNDS/OptMPNDS2: multiparty-aware nondominated sorting algorithms
• MPMOEA-MOEA/D: a decomposition-based multiparty extension of MOEA/D
• Multiparty SPEA2 variants
• Privacy-preserving and communication-limited multiparty extensions
• Biparty-specific MPMOEAs for power-flow, distance minimization, and related application domains

All parameterizations, stop criteria, and tuning details are left open for research innovation.

4. Limitations and Future Directions

The MPBench MPMOP suite is the first standardized test suite for multiparty optimization. Current limitations and agenda for further work include:

• Enabling more than two or three DMs, with heterogeneous privacy or communication arrangements.
• Time-dependent (dynamic) multiparty objectives beyond those in the constructed analytic set.
• Benchmarking on genuine large-scale, high-dimensional multiparty optimization problems arising in real-world domains (e.g., supply chain, smart grids, multiagent systems).
• Development of privacy-preserving and communication-efficient evolutionary algorithms for the multiparty setting.
• Theoretical investigation into the geometry and convergence behavior of multiparty Pareto fronts (Luo et al., 2024).

5. Related “MPBench” Benchmarks in Other Fields

• Multimodal Reasoning:
  • MPBench in (Xu et al., 16 Mar 2025) designates a benchmark for process-level reward models (PRMs) in chain-of-thought reasoning, spanning error identification, answer aggregation, and process search across text and image modalities.
• Model Perception (Image Forgery Detection):
  • In (Lu et al., 2023), MPBench is a testbed for evaluating fake-image detection models on the Fake2M dataset, focusing on classification accuracy and failure rates over diverse synthetic image generators, and benchmarks both models and human performance.
• Mobile Device ML Inference:
  • As the MLPerf Mobile Inference Benchmark, MPBench (Reddi et al., 2020) measures performance and accuracy of mobile AI stacks across hardware, operator APIs, and model zoo tasks, using end-to-end measurement and strict reproducibility protocols.

6. Significance and Impact

MPBench, in its various incarnations, addresses systematic gaps in the evaluation of model, algorithm, or system-level performance in settings that demand multimodality, multiparty decision-making, or real-world operational constraints. The CEC 2024 competition version fills the foundational gap for multiparty multiobjective optimization, providing clear definitions, well-documented analytic and realistic test cases, and uniformly adopted multiparty metrics. By creating public, reproducible evaluation infrastructure, MPBench-family benchmarks catalyze algorithmic comparison, foster methodological advances, and enable the rigorous demonstration of progress across each of their domains (Luo et al., 2024, Xu et al., 16 Mar 2025, Lu et al., 2023, Reddi et al., 2020).