MLPerf Training v4.1 Benchmark Overview

• MLPerf Training v4.1 is a benchmark suite designed to evaluate end-to-end deep learning training performance with fixed quality targets and controlled variance.
• It introduces a recommendation workload that bridges classic vision/NLP tasks and commercial systems by evaluating both candidate generation and ranking.
• The framework employs rigorous methods such as repeated executions, exclusion of extreme runs, and detailed performance metrics to ensure fair hardware and software comparisons.

Searching arXiv for MLPerf Training v4.1 and related benchmark papers to ground the article in the cited literature. MLPerf Training v4.1 is a community-standard suite of training workloads designed to measure end-to-end system performance and convergence to fixed quality targets, extending MLPerf’s formal time-to-train methodology with refreshed workloads and, in v4.1, a Recommendation benchmark intended to fill the gap between classic vision/NLP tasks and large-scale commercial recommender systems (Mattson et al., 2019, Wu et al., 2020). Its design reflects three benchmark-specific difficulties emphasized in the MLPerf literature: optimizations that increase raw throughput can degrade convergence, large-scale parallel training changes optimization dynamics, and stochastic variation can obscure genuine system effects (Mattson et al., 2019). The resulting framework combines reference implementations, submission rules, variance controls, and workload-specific quality thresholds so that hardware and software systems can be compared under a common methodology.

1. Historical lineage and benchmark purpose

MLPerf Training emerged from the need for an industry-standard performance benchmark for machine learning training that remains meaningful despite stochastic optimization, heterogeneous systems, and model-dependent convergence behavior. The foundational formulation defines the primary metric as time to solution, or “time-to-train,” rather than raw throughput alone. If a system produces a validation-accuracy curve $a(t)$ and the task has a target quality threshold $A_{\text{target}}$, then the time to solution is

$$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$

This timing begins from the first read of any training or validation data and ends when the threshold is reached (Mattson et al., 2019).

The same framework also formalizes repeated execution to control stochastic variability. Each benchmark must be executed $N$ times, with $N=5$ for most vision tasks and $N=10$ for other tasks; MLPerf drops the fastest $k$ and slowest $k$ runs, with $k=1$ by default, and reports the arithmetic mean over the remaining runs. The standard deviation is also monitored so that submissions from the same system lie within $\pm 5\%$ for vision or $A_{\text{target}}$0 for other tasks (Mattson et al., 2019).

Within this lineage, v4.1 is characterized in the supplied literature in two complementary ways. First, it continues the benchmark’s general role as the de-facto standard for measuring deep-learning training performance, while refreshing the suite to cover emerging motifs such as LLMs and generative diffusion networks (Mattson et al., 2019). Second, it explicitly adds a Recommendation workload whose stated purpose is to represent both the latency-sensitive “candidate generation” stage and the compute-heavy “ranking” stage of commercial recommender systems (Wu et al., 2020). This suggests that v4.1 is not merely an incremental revision of legacy workloads, but a suite in which benchmark coverage is being actively rebalanced toward current deployment patterns.

2. Measurement methodology, divisions, and reporting constraints

MLPerf does not reduce benchmark results to a single aggregate score. Instead, results tables present the benchmark name and quality threshold, the system configuration, the division and category, the reported mean time $A_{\text{target}}$1, and optionally power draw for on-premise systems or a cloud-scale index correlated with cost (Mattson et al., 2019). This reporting design is a direct response to the absence of any universally agreed weighting across tasks and to the fact that some systems may choose not to run every benchmark.

The benchmark distinguishes between a Closed Division and an Open Division. In the Closed Division, submissions must strictly follow the reference implementation’s model topology, data-processing pipeline, parameter initialization, and learning-rate schedule, except for a controlled set of modifiable hyperparameters such as batch size, learning-rate scale factors, and optimizer parameters. No changes to network architecture, loss function, data-augmentation orders, or major algorithmic tricks are allowed. In the Open Division, architectures, optimizer schedules, precision formats, communication algorithms, and related design choices may be changed freely (Mattson et al., 2019). This distinction is central to MLPerf’s claim of fairness: direct vendor-to-vendor comparison is tied to the closed rules, whereas open submissions emphasize hardware-software co-design innovation.

All submissions must provide full source, container recipe, and build scripts; declare system details including accelerator count, memory sizes, interconnect topology, operating system, framework versions, and math libraries; and supply structured logs with timestamps and accuracy reports (Mattson et al., 2019). Categories further label systems as Available, Preview, or Research, depending on shipping status or experimental character (Mattson et al., 2019).

A common misconception is that training benchmarks are fundamentally throughput benchmarks. MLPerf’s formalism rejects that equivalence: throughput can improve while time to solution worsens, because lower precision, communication changes, or very large minibatches may alter convergence. The benchmark is therefore structured around quality-constrained completion rather than unconstrained processing rate (Mattson et al., 2019).

3. Workload composition and the evolution toward v4.1

The benchmark lineage summarized in the MLPerf literature includes seven representative training tasks: Image Classification with ResNet-50 v1.5 on ImageNet; Object Detection and Segmentation with SSD-ResNet-34 and Mask R-CNN on COCO; Neural Machine Translation with GNMT and Transformer; Recommendation with Neural Collaborative Filtering on MovieLens-20M; and Reinforcement Learning with MiniGo (Mattson et al., 2019). These tasks were chosen to span convolutional, attention, recurrent, sparse-embedding, and reinforcement-learning workloads while remaining compact enough to run on a wide variety of systems (Mattson et al., 2019).

The v4.1-specific materials indicate both continuity and change. One analysis of MLPerf Training v4.1 results studies four workloads—BERT, Llama2 LoRA, RetinaNet, and Stable Diffusion—and reports hardware configurations ranging from small systems to clusters with thousands of GPUs, along with efficiency calculations derived from the official MLCommons v4.1 code repositories and config files (Cortes et al., 3 Sep 2025). Separately, the recommendation-benchmark proposal states that v4.1 adds a Recommendation workload to fill the gap between classic vision/NLP tasks and large-scale commercial recommender systems (Wu et al., 2020).

These sources jointly indicate that v4.1 is best understood as a benchmark generation organized around methodological continuity rather than a fixed, once-for-all workload list. The literature explicitly notes that the consortium continues to refresh the suite to cover emerging motifs, and the presence of BERT, Llama2 LoRA, RetinaNet, Stable Diffusion, and Recommendation in the supplied v4.1 materials is consistent with that stated direction (Mattson et al., 2019, Cortes et al., 3 Sep 2025).

4. Recommendation benchmark in MLPerf Training v4.1

The Recommendation workload introduced in v4.1 is defined as a “medium-scale” recommender that balances realism with practicality. Its prescribed scale includes 10–100 sparse embedding tables with 10 K–10 M categories each, 50–200 dense features, an embedding dimension $A_{\text{target}}$2, and a tower or “wide-and-deep” architecture with at most 10 hidden layers in each MLP (Wu et al., 2020). The benchmark’s twin goals are to represent both candidate generation and ranking, and to drive fair comparison across hardware/software stacks for both training throughput and end-to-end inference latency (Wu et al., 2020).

A typical two-tower Deep Learning Recommendation Model (DLRM) is adopted. For the $A_{\text{target}}$3-th categorical feature of sample $A_{\text{target}}$4, sparse lookup is written as

$A_{\text{target}}$5

Dense real-valued features $A_{\text{target}}$6 are processed by an $A_{\text{target}}$7-layer MLP,

$A_{\text{target}}$8

with ReLU nonlinearity. Optional feature interactions compute pairwise inner products between embedding vectors and/or dense features and stack them into an interaction vector $A_{\text{target}}$9. The final prediction concatenates $$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$0 and applies a dense layer plus sigmoid,

$$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$1

Training uses binary cross-entropy / logistic loss,

$$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$2

All of these components are explicitly identified in the v4.1 recommendation summary (Wu et al., 2020).

The principal dataset rationale centers on the Criteo Kaggle CTR-terabyte dataset, described as approximately 45 M examples with 13 dense features and 26 categorical features hashed to 10 M–20 M buckets. Its one-hot categorical representation has sparsity greater than $$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$3, and its power-law distribution of category frequencies is said to mirror production ad-serving workloads (Wu et al., 2020). MovieLens is deemed too simple and is dropped for Training v4.1, while Fractal-expanded MovieLens and Criteo-plus variants inform future “large”-scale tracks (Wu et al., 2020).

Evaluation uses both binary-classification and ranking metrics. The summary gives

$$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$4

and

$$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$5

Training throughput is measured as

$$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$6

while inference latency is the 99th-percentile time to process a single query with batch size $$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$7 (Wu et al., 2020). The v4.1 target thresholds given as an example for an 8$$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$8V100 GPU system are training at at least 200 k samples/sec sustained and inference at at most 0.3 ms tail latency at 1 QPS per GPU, with exact numbers varying by submission category (Wu et al., 2020).

The recommended hyperparameters and system guidelines include default embedding dimension $$T_s = \min \{ t \mid a(t) \ge A_{\text{target}} \}.$$9, tunable to $N$0 per table; a dense MLP tower of 3–5 layers with sizes $N$1 and ReLU activations; SGD with momentum 0.9 or Adam; initial learning rate 0.1 with linear warm-up over 1 epoch and cosine decay; batch size 65,536 samples per GPU; gradient accumulation across multiple GPUs or nodes; large CPU host memory of at least 128 GB; host-device bandwidth of at least 64 GB/s via NVLink or PCIe-Gen4; overlapped sparse-embedding lookups and dense GEMMs; fused kernels such as FBGEMM and cuDNN; and PyTorch or TensorFlow with custom C++/CUDA kernels for hashing, embedding, interaction, and fused MLP operations (Wu et al., 2020).

5. Software stacks, scaling behavior, and efficiency in v4.1 studies

A detailed study of MLPerf Training v4.1 scalability reports hardware spanning NVIDIA H100-SXM5-80 GB, NVIDIA Blackwell, NVIDIA H200, NVIDIA RTX series, and TPU v5p, with workload-specific GPU counts ranging from 2 up to 3,472 for BERT, 4 up to 1,024 for Llama2 LoRA, 4 up to 2,528 for RetinaNet, and 4 up to 1,024 for Stable Diffusion (Cortes et al., 3 Sep 2025). Typical interconnects are NVLink-HDR, InfiniBand HDR200 or HDR400, PCIe 5.0 in smaller systems, and Google’s proprietary high-speed mesh in some TPU v5p pods (Cortes et al., 3 Sep 2025).

The reported software stack includes PyTorch 2.0 or TensorFlow 2.11 with HuggingFace and DeepSpeed or LoRA patches for BERT and Llama2-LoRA, PyTorch 1.13 with torchvision for RetinaNet, and PyTorch 2.0 with the LDM codebase for Stable Diffusion. Communication libraries include NCCL 2.x and Horovod 0.26. All runs in that study use the closed division, with mlperf-compliance enabled, and use the official MLCommons v4.1 code repositories and config files (Cortes et al., 3 Sep 2025).

The same study defines speedup $N$2, parallel efficiency $N$3, and adjusted per-GPU efficiency $N$4 as

$N$5

and

$N$6

where $N$7 is the reference time on $N$8 GPUs (Cortes et al., 3 Sep 2025). It also defines a “break-even” configuration $N$9 as the smallest $N=5$0 for which adding one more GPU yields no net gain in throughput per dollar or per energy unit, expressed as $N=5$1 (Cortes et al., 3 Sep 2025).

The reported general trends are consistent across the four studied workloads: up to approximately 8 GPUs, scaling is near-ideal with $N=5$2; from 8 to 16 GPUs, efficiency drops to roughly 0.6–0.7; beyond 16 GPUs, diminishing returns become rapid, and by hundreds or thousands of GPUs, $N=5$3 falls below 0.1 (Cortes et al., 3 Sep 2025).

Workload	Observed GPU range	Approximate break-even point
BERT	2–3,472	12–16 GPUs
Llama2-LoRA	4–1,024	8–12 GPUs
RetinaNet	4–2,528	16–20 GPUs
Stable Diffusion	4–1,024	10–12 GPUs

The study further reports workload-specific interpretations: BERT and Stable Diffusion scale “reasonably” to approximately 16 GPUs before efficiency penalties exceed 30–40%; Llama2-LoRA experiences the steepest decline because of large parameter count and frequent gradient exchanges; and RetinaNet retains better linearity out to approximately 16 GPUs because its backbone pipelining reduces synchronization points (Cortes et al., 3 Sep 2025). A common misconception is therefore that more accelerators necessarily improve the practical quality of a training deployment. The v4.1 evidence indicates instead that strong scale-out and resource efficiency diverge sharply beyond a workload-specific “sweet spot.”

6. Compression research, benchmark interpretation, and broader significance

The recommendation-oriented compression work “Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model” uses the v4.1 CriteoTB DLRM benchmark to examine an especially important systems bottleneck: large embedding tables (Desai et al., 2021). In the official reference implementation summarized there, the benchmark uses the 23-day “Terabyte” Criteo click-through-rate dataset with 13 dense features and 26 categorical fields; the embedding tables occupy about 100 GB of GPU memory, the model converges to ROC-AUC 0.8025 in exactly one pass through the data, and peak inference throughput is approximately 0.34 million samples/sec on four Quadro RTX 8000 GPUs (Desai et al., 2021).

ROBE replaces large embedding tables with a single compact circular array $N=5$4, using hash functions to map embeddings or embedding blocks into that array. The paper states that the inner-product estimator after ROBE-Z projection is unbiased and that increasing block size $N=5$5 strictly lowers projection variance (Desai et al., 2021). On CriteoTB, the reported comparison is from 100 GB embedding memory to 0.1 GB, convergence from 1 epoch to 1.89 epochs to reach AUC $N=5$6, final AUC from 0.8031 to 0.8025, and inference throughput from 341,454 samples/sec on the original configuration to 1,055,470 samples/sec for ROBE-128 (Desai et al., 2021). The same source attributes the speed effects to cache behavior, coalesced accesses, and the elimination of model parallelism (Desai et al., 2021).

These results are significant for interpreting v4.1 because they sharpen a broader point already present in MLPerf’s design: benchmark performance is not reducible to a single resource axis. In the general benchmark methodology, throughput can mislead when convergence changes (Mattson et al., 2019); in the v4.1 scaling study, aggressive GPU count can degrade per-GPU efficiency (Cortes et al., 3 Sep 2025); and in the recommendation benchmark, sparse lookups and dense neural operators must both be optimized to meet throughput and latency thresholds (Wu et al., 2020). ROBE adds a further dimension by showing that memory-footprint reduction can alter the feasible parallelization regime itself (Desai et al., 2021).

A plausible implication is that MLPerf Training v4.1 functions less as a static scorecard than as an experimental protocol for exposing trade-offs among convergence, latency, memory movement, communication, and accelerator utilization. That interpretation is consistent with the benchmark’s formal reliance on fixed quality targets, its closed-division reproducibility rules, its refusal to publish a single aggregate score, and its ongoing expansion toward workloads that stress different parts of the modern AI systems stack (Mattson et al., 2019).