---
title: 'MPBench: Multiparty Optimization & Evaluation'
url: https://www.emergentmind.com/topics/mpbench
type: topic
---

# MPBench: Multiparty Optimization & Evaluation

MPBench is a designation held by several distinct benchmark suites across different research domains, each serving as a rigorous, open-source evaluation standard for models or algorithms at the forefront of their respective fields. The following survey catalogs key MPBench instances: (1) as a comprehensive testbed for process-level reward models in multimodal reasoning, (2) as a model perception benchmark for detecting AI-generated images, (3) as the official benchmark for multiparty multiobjective optimization at the CEC 2024 competition, and (4) as the MLPerf Mobile Inference Benchmark (commonly abbreviated MPBench) for cross-platform device ML evaluation. Each version is independent, and the label "MPBench" must be contextualized by its research community.

## 1. Multiparty Multiobjective Optimization: MPBench for CEC 2024

MPBench for CEC 2024 formalizes the multiparty multiobjective optimization problem (MPMOP) paradigm, in which $M$ distinct decision makers (DMs) each possess vector-valued objective functions, and a solution is considered multiparty-optimal if no feasible candidate is weakly better for all DMs and strictly better for at least one. Formally, the MPMOP is:

\[
\min\;E(\mathbf x)\;=\;\bigl[F_1(\mathbf x),\,F_2(\mathbf x),\dots,F_M(\mathbf x)\bigr],
\]
with $F_i(\mathbf x) = (f_{i1}(\mathbf x), \dots, f_{im_i}(\mathbf x))$ representing the objective suite for DM $i$.

### Benchmark Structure

MPBench divides its suite into two major classes:

- **Part I: Analytic Test Problems with Known Pareto Sets**
  - 11 MPMOPs, derived by combining time-indexed dynamic MOP functions (BF1–BF6 from CEC’2018, via Liu et al. 2020). Each problem incorporates multiple DMs (either pairs or triplets), each assigned objective surfaces at different time-slices ($t$). All share a nonempty analytical Pareto-optimal set.
  - Example construction:

    \[
    E_1(\mathbf x) = \bigl[ F_{11}(\mathbf x), F_{12}(\mathbf x) \bigr],
    \]
    where $F_{11}(\mathbf x) = [ f_{11}(\mathbf x, 1), f_{12}(\mathbf x, 1) ]$ and $F_{12}(\mathbf x) = [ f_{11}(\mathbf x, 2), f_{12}(\mathbf x, 2) ]$.
  - Problem dimensions: $n \in \{10, 30, 50\}$.

- **Part II: Realistic Path Planning Scenarios**
  - Six biparty UAV path planning (BP-UAVPP) problems, in which DMs represent efficiency (minimizing trajectory cost, fuel, height change, and distance to hover-points) and safety (minimizing fatality risk, property risk, noise), over discrete trajectory waypoints under combined terrain and vehicle constraints. Objective formulas, such as $f_{\text{fuel}}$ and $f_{\text{fatal}}$, are given precisely in the benchmark description.
  - Six scenarios ($C_1$–$C_6$) vary in assignment and inclusion of objectives for each party.

## 2. Evaluation Metrics and Competition Protocol

- **Part I evaluation (Pareto known):** Multiparty Inverted Generational Distance (MPIGD):

  \[
  \operatorname{MPIGD}(P^{MP},P) = \frac{1}{|P^{MP}|}\sum_{v \in P^{MP}} d(v, P),
  \]
  with
  \[
  d(v, P) = \min_{s \in P} \sum_{j=1}^M \sqrt{ \sum_{\ell=1}^{m_j} (v_{j\ell} - s_{j\ell})^2 }.
  \]

- **Part II evaluation (Pareto unknown):** Multiparty Hypervolume (MPHV), summed or averaged across parties:

  \[
  \mathrm{MPHV} = \frac{1}{M} \sum_{i=1}^M HV_i,
  \]
  where $HV_i$ is the individual hypervolume achieved by DM $i$’s solution set.

- **Ranking:** The final benchmark score is the average algorithm rank across all problems (all 17 test cases).

- **Competition protocol:** Dimension, maximum function evaluations ($\mathrm{FE}_{\max}$), and number of independent runs (30) are fixed; all other details (algorithm choice, population sizes, parameter settings) are at the discretion of contestants. No baseline or comparative results are supplied in the benchmark paper; these will be established post-competition by aggregating MPIGD/MPHV metrics and rankings [2402.02033].

## 3. Reference Algorithms and Literature Context

While MPBench itself does not prescribe baselines or report empirical results, it catalogs established approaches such as:

- OptMPNDS/OptMPNDS2: multiparty-aware nondominated sorting algorithms
- MPMOEA-MOEA/D: a decomposition-based multiparty extension of MOEA/D
- Multiparty SPEA2 variants
- Privacy-preserving and communication-limited multiparty extensions
- Biparty-specific MPMOEAs for power-flow, distance minimization, and related application domains

All parameterizations, stop criteria, and tuning details are left open for research innovation.

## 4. Limitations and Future Directions

The MPBench MPMOP suite is the first standardized test suite for multiparty optimization. Current limitations and agenda for further work include:

- Enabling more than two or three DMs, with heterogeneous privacy or communication arrangements.
- Time-dependent (dynamic) multiparty objectives beyond those in the constructed analytic set.
- Benchmarking on genuine large-scale, high-dimensional multiparty optimization problems arising in real-world domains (e.g., supply chain, smart grids, multiagent systems).
- Development of privacy-preserving and communication-efficient evolutionary algorithms for the multiparty setting.
- Theoretical investigation into the geometry and convergence behavior of multiparty Pareto fronts [2402.02033].

## 5. Related “MPBench” Benchmarks in Other Fields

- **Multimodal Reasoning:**
  - MPBench in [2503.12505] designates a benchmark for process-level reward models (PRMs) in chain-of-thought reasoning, spanning error identification, answer aggregation, and process search across text and image modalities.
- **Model Perception (Image Forgery Detection):**
  - In [2304.13023], MPBench is a testbed for evaluating fake-image detection models on the Fake2M dataset, focusing on classification accuracy and failure rates over diverse synthetic image generators, and benchmarks both models and human performance.
- **Mobile Device ML Inference:**
  - As the MLPerf Mobile Inference Benchmark, MPBench [2012.02328] measures performance and accuracy of mobile AI stacks across hardware, operator APIs, and model zoo tasks, using end-to-end measurement and strict reproducibility protocols.

## 6. Significance and Impact

MPBench, in its various incarnations, addresses systematic gaps in the evaluation of model, algorithm, or system-level performance in settings that demand multimodality, multiparty decision-making, or real-world operational constraints. The CEC 2024 competition version fills the foundational gap for multiparty multiobjective optimization, providing clear definitions, well-documented analytic and realistic test cases, and uniformly adopted multiparty metrics. By creating public, reproducible evaluation infrastructure, MPBench-family benchmarks catalyze algorithmic comparison, foster methodological advances, and enable the rigorous demonstration of progress across each of their domains [2402.02033, 2503.12505, 2304.13023, 2012.02328].

Source: https://www.emergentmind.com/topics/mpbench