MLPerf Inference Benchmark Overview

• MLPerf Inference Benchmark is a community-driven suite that provides architecture-neutral and reproducible metrics for evaluating machine learning inference across edge, cloud, mobile, and datacenter environments.
• It employs a scenario-based evaluation with four distinct real-world deployment patterns—single-stream, multistream, server, and offline—to capture latency, throughput, and energy efficiency metrics.
• The benchmark drives practical insights in hardware-software co-design by incorporating methods like quantization, batching, and model compression to optimize performance across diverse workloads.

The MLPerf Inference Benchmark is the primary community-driven, architecture-neutral suite for evaluating the performance, accuracy, and efficiency of machine learning inference systems across the full spectrum of hardware, software, and application domains. Since its introduction, it has defined the de facto experimental standards and reporting formats for ML inference across edge, cloud, mobile, and datacenter environments. MLPerf Inference underpins industry and academic efforts to quantify progress in real-world deployment, facilitating fair, reproducible comparison and rapid integration of new workloads, models, and optimization strategies (Reddi et al., 2019).

1. Benchmark Rationale and Core Principles

MLPerf Inference was created to address the combinatorial diversity and lack of comparability in ML hardware and software evaluation. It responds to the proliferation of inference chips, frameworks, and system architectures, whose metrics (throughput, latency, power, and accuracy) were previously reported in incompatible, scenario-specific ways. The benchmark’s explicit goal is representative, architecture-neutral, reproducible, and statistically rigorous inference performance evaluation, spanning embedded microcontrollers up to hyperscale accelerators, via a unifying methodology and transparent ruleset (Reddi et al., 2019).

Targets for MLPerf Inference include vision, language, and recommendation workloads on consumer, edge, and enterprise systems. These include real-time AR, camera pipelines (single- and multi-stream), edge robotics, cloud-scale translation, and large-scale recommendation serving (Reddi et al., 2019, Wu et al., 2020).

2. Scenario-Based Evaluation Methodology

MLPerf Inference anchors its design around four canonical scenarios, which mimic distinct real-world deployment patterns and guide the interpretation of model/system performance:

Scenario	Query Pattern	Main Metric	Use Case Example
Single-stream	Sequential, 1/req	90th-percentile latency	Real-time typing, AR, speech
Multistream	Fixed parallelism	Max streams within latency	Multi-camera driver assist
Server	Poisson arrivals	QPS under latency bound	Online translation web API
Offline	Unbounded batch	Throughput, samples/sec	Batch image/video labeling

Each scenario enforces precise statistical measurement protocols. For example, offline and multistream scenarios require ≥24,576 or 270,336 queries, respectively, with stringent confidence intervals for latency percentiles (e.g., p99 with <0.05% error, to 99% confidence) (Reddi et al., 2019). This statistical rigor ensures that tail-latency figures and throughput numbers are reliable and comparable across diverse systems.

Metrics are defined as follows:

• Throughput: $$\mathrm{Throughput} = \frac{\text{Total samples processed}}{\text{Total wallclock time}}$$ (e.g., images/s, tokens/s).
• Latency percentiles: $L_p$ such that $p\%$ of requests are completed within $L_p$.
• Accuracy: Task-specific (e.g., top-1 accuracy, mAP, BLEU, ROUGE), specified as a fraction of the FP32 reference.
• Time-to-first-token (TTFT) (LLM/server scenario): $$\mathrm{TTFT}_i = t_i^{\mathrm{first\,token}} - t_i^{\mathrm{issue}}$$.
• Energy: $$E_{\text{total}} = \int_{t_0}^{t_{\text{end}}} P(t)\,dt$$ (Joules), or normalized as Joules/sample.
• Cost (deployment-centric): $C_{\mathrm{token}} = \frac{C_{\$/\mathrm{hr}}}{T \times 3600}$$(<a href="/papers/2509.11413" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Fursin et al., 14 Sep 2025</a>).</li> </ul> <h2 class='paper-heading' id='workload-suite-models-and-compliance-rules'>3. Workload Suite, Models, and Compliance Rules</h2> <p>The MLPerf Inference suite encompasses a diverse array of models, tasks, and datasets, all with public reference implementations and open-source harnesses:</p> <ul> <li><strong>Vision</strong>: ResNet-50 v1.5, MobileNet-v1/EdgeTPU, SSD-ResNet-34/SSD-MobileNet, DeepLab v3+ (classification, detection, segmentation) (<a href="/papers/1911.02549" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Reddi et al., 2019</a>, <a href="/papers/2012.02328" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Reddi et al., 2020</a>).</li> <li><strong>Language</strong>: GNMT, BERT, MobileBERT, LLMs (ROUGE/BLEU score) (<a href="/papers/2509.11413" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Fursin et al., 14 Sep 2025</a>, <a href="/papers/2012.02328" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Reddi et al., 2020</a>).</li> <li><strong>Recommendation</strong>: DLRM-class two-tower models with large sparse embeddings, typically evaluated on Criteo Kaggle/MLPerf CriteoTB (<a href="/papers/2003.07336" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Wu et al., 2020</a>, <a href="/papers/2108.02191" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Desai et al., 2021</a>).</li> <li><strong>Edge/Mobile/Tiny</strong>: Keyword spotting, anomaly detection, “tiny” image models, with int8/fixed-point quantization and$$\mu$s- to ms-level latency (Reddi et al., 2020, Borras et al., 2022).
• LLM Inference: Full-stack LLM benchmarks, e.g., LLaMA, DeepSeek, with standardized latency, throughput, and accuracy extraction (ROUGE metrics) (Fursin et al., 14 Sep 2025).

Compliance operates in two divisions:

• Closed division: Only allow specified quantization, graph transform, or software optimization; must use MLPerf reference weights and datasets.
• Open division: Permits model changes, custom weights, or architectural innovations—intended for showcasing novel optimization techniques (e.g., 4-bit quantization, novel architectures).

Mandatory submission artifacts include full system specification, software stack, quantization/calibration method, accuracy logs, performance logs, and automated validation outputs.

4. Modular Architecture and Automation Frameworks

The MLPerf Inference workflow is modular, built around core building blocks:

• LoadGen: The canonical C++/Python query generator and metric extractor, supporting all four scenarios, strict arrival patterns, and standardized logging (Reddi et al., 2019, Fursin et al., 14 Sep 2025).
• Harnesses and Inference Engines: MLPerf harnesses abstract model loading, preprocessing, and prediction via standardized APIs. Frameworks include TensorFlow, ONNX Runtime, PyTorch, TFLite, OpenVINO, ArmNN, vendor SDKs, and custom FPGA toolchains (Reddi et al., 2020, Ahn et al., 2023, Borras et al., 2022).
• Automation and Orchestration: CM4MLPerf/Collective Mind (CM) provides unified CLI/Python automation scripts for reproducible end-to-end runs: environment detection, model/dataset acquisition, dependency management, LoadGen invocation, measurement logging, and submission packaging. Modular “script” interfaces permit extension to new hardware/models with minimal effort (Fursin, 2024).
• FlexBench and Open MLPerf Dataset: Recent extensions (FlexBench) wrap LoadGen with plug-in filters, HuggingFace integration, and a test harness for continuous, collaborative dataset expansion. This enables community-driven, extensible benchmarking and predictive modeling based on large-scale public results databases (Fursin et al., 14 Sep 2025).

5. Data Organization, Result Reporting, and Analysis

MLPerf results are stored as structured JSON (or JSON-Lines), supporting both official and community-generated submissions. The minimal schema includes fields for metrics, hardware, software, and provenance:

Field	Type	Example/Description
metrics.accuracy	string	“ROUGE1:30.62 ROUGE2:13.92 ...”
metrics.result	float	tokens/s or images/s
model.mlperf_name	string	“llama2-70b-99”
system.accelerator.name	string	“NVIDIA H100 80GB HBM3”
software.framework	string	“vLLM v0.7.3”
submission.scenario	string	“Server” or “Offline”
...	...	...

The result aggregation pipeline routinely consists of the following steps: (1) automated benchmark execution, (2) metric/metadata extraction, (3) normalization and feature engineering (e.g., one-hot encoding for hardware or precision), (4) distribution via open datasets and dashboards (e.g., FlexBoard in Gradio for interactive exploration) (Fursin et al., 14 Sep 2025).

Quantitative results illustrate extensive device/model coverage: e.g., benchmark results for LLMs on NVIDIA H100 achieving throughputs >2,400 tokens/s at low cost per token, recommender models compressing embeddings by 1,000$\times$ with 3.1$\times$ higher throughput, and MLPerf Tiny FPGA entries with $<20\,\mu$s latency per inference (Fursin et al., 14 Sep 2025, Desai et al., 2021, Borras et al., 2022).

6. Optimization Practices and Hardware/Software Co-design

MLPerf Inference actively encourages hardware–software co-design, with target metrics tailored to different system bottlenecks:

• Quantization: Support for post-training and quantization-aware int8/fp16/bfloat16 transformations, with calibration on MLPerf’s small subset. For edge/ARM, static int8 quantization (TFLite, OpenVINO's INT8-OM) is optimal, yielding 3–4$\times$ throughput increases (Ahn et al., 2023).
• Batching and Parallelism: Batch sizes and per-instance parallelism are tuned subject to scenario queueing rules (e.g., batch 32 for x86 CPUs, batch 4–8 for embedded ARM) for throughput maximization (Ahn et al., 2023, Castelló et al., 2021).
• Model Compression: Embedding table compression (e.g., ROBE for DLRM) reduces memory use 1,000$\times$ with negligible accuracy impact and 3.1$\times$ inference speedup (Desai et al., 2021). FPGA codesign with extreme quantization (e.g., 1–3 bit activations) achieves $<30\,\mu$J energy/inference (Borras et al., 2022).
• System-Level Optimizations: Layer fusion, microkernel vectorization (NEON/AVX), explicit cache blocking, dataflow scheduling, and data orchestration primitives are crucial on both CPUs and accelerators (Castelló et al., 2021, Davies et al., 2021).
• Automation: Unified orchestration (CM, FlexBench) reduces manual overhead, ensures reproducibility, and accelerates benchmarking cycles (Fursin, 2024, Fursin et al., 14 Sep 2025).

Notably, unconstrained offline throughput can be a poor predictor of latency-constrained server performance (throughput reductions of 3–50% are observed in the latter), emphasizing the importance of scenario fidelity (Reddi et al., 2019).

7. Community Practices, Extensions, and Future Trajectory

MLPerf Inference is an evolving suite—its modular LoadGen and harness interfaces, scenario definitions, and flexible submission structure allow rapid addition of new tasks (e.g., RNN-T, super-resolution), modalities, and hardware backends. Community-led datasets (Open MLPerf Dataset, FlexBoard) and automation (CM4MLOps, Gradio dashboards) enable efficient cost/latency/accuracy trade-offs tailored to deployment constraints (Fursin et al., 14 Sep 2025, Fursin, 2024).

Planned future directions encompass more realistic, composite workloads (end-to-end pipelines, federated/heterogeneous systems), power/energy benchmarking under operational loads, and predictive modeling of deployment trade-offs using large-scale crowdsourced result sets. Modular, “learning loop” workflows help integrate new hardware, models, and system-level meta-optimizers, driving MLPerf’s continued relevance as the field evolves.

---

References:

• “MLPerf Inference Benchmark” (Reddi et al., 2019)
• “Framing AI System Benchmarking as a Learning Task: FlexBench and the Open MLPerf Dataset” (Fursin et al., 14 Sep 2025)
• “Performance Characterization of using Quantization for DNN Inference on Edge Devices: Extended Version” (Ahn et al., 2023)
• “Open-source FPGA-ML codesign for the MLPerf Tiny Benchmark” (Borras et al., 2022)
• “Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model” (Desai et al., 2021)
• “Developing a Recommendation Benchmark for MLPerf Training and Inference” (Wu et al., 2020)
• “Violet: Architecturally Exposed Orchestration, Movement, and Placement for Generalized Deep Learning” (Davies et al., 2021)
• “MLPerf Mobile Inference Benchmark” (Reddi et al., 2020)
• “High performance and energy efficient inference for deep learning on ARM processors” (Castelló et al., 2021)
• “Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments” (Fursin, 2024)