MLPerf Automotive Benchmark

• MLPerf Automotive is a benchmark suite tailored for evaluating ML systems in vehicles, emphasizing safety, determinism, and strict tail-latency metrics.
• It standardizes three perception tasks—2D object detection, 2D semantic segmentation, and camera-only 3D object detection—using high-resolution datasets like MLCommons Cognata and nuScenes.
• The benchmark employs common reference models in ONNX format and LoadGen integration to ensure reproducible, system-level comparisons under realistic automotive constraints.

Searching arXiv for the specified papers to ground the article in current literature. I’ll look up the cited arXiv entries and use them as the primary sources. MLPerf Automotive is a benchmark suite for evaluating machine learning systems deployed for AI acceleration in automotive systems under constraints that matter in vehicles, notably safety, determinism, and real-time latency. Developed jointly by MLCommons and the Autonomous Vehicle Computing Consortium, it is described as the first standardized public benchmark for this domain and adopts the broad structure of MLPerf Inference—common LoadGen, reference implementations, closed and open divisions, and accuracy and compliance checks—while tailoring workloads, datasets, scenarios, and metrics to driving workloads (Shojaei et al., 31 Oct 2025). The first release, v0.5 in July 2025, standardizes three perception tasks—2D object detection, 2D semantic segmentation, and camera-only 3D object detection—using high-resolution inputs, 99.9% tail-latency targets, and accuracy measured as retention relative to FP32 reference baselines.

1. Automotive benchmarking context and design rationale

MLPerf Automotive was introduced because automotive ML workloads differ materially from datacenter, mobile, and IoT workloads. The motivating constraints are safety-critical operation, continuous real-time execution, and large, heterogeneous sensory inputs ranging from dense camera arrays to sparse LiDAR and Radar under tight latency budgets. The benchmark description also emphasizes long automotive system lifecycles—9–23 years for vehicles—and the role of functional safety and qualification requirements such as ISO 26262 and AEC-Q100 in shaping hardware and software design (Shojaei et al., 31 Oct 2025).

These constraints change what constitutes a meaningful performance comparison. In particular, many automotive workloads must be deterministic with bounded jitter because actuation decisions depend on tail behavior rather than only mean latency. MLPerf Automotive therefore preserves the comparative discipline of MLPerf Inference while modifying the operational target: performance is characterized primarily through 99.9% percentile tail latency under standardized scenarios, with accuracy retention constraints imposed relative to FP32 reference models. This design makes the suite less a generic perception benchmark than a methodology for reproducible system-level comparison under automotive operating assumptions.

2. Benchmark scope, tasks, and datasets

The v0.5 suite targets representative perception workloads across the SAE autonomy spectrum. It includes single-camera 2D object detection, single-camera 2D semantic segmentation, and multi-camera camera-only 3D object detection. The paper explicitly uses the MLCommons Cognata synthetic driving dataset for the 2D tasks and nuScenes for the 3D task; KITTI, Cityscapes, Waymo Open, Mapillary Vistas, and ApolloScape appear only as related work and context (Shojaei et al., 31 Oct 2025).

Workload	Reference model	Dataset and input
2D object detection	SSD with ResNet-50 backbone	MLCommons Cognata; 1 image at 3840×2160
2D semantic segmentation	DeepLabv3+ with ResNet-50 backbone	MLCommons Cognata; 1 image at 3840×2160
3D object detection	BEVFormer-tiny with ResNet-50 backbone	nuScenes; 6 images per query at 800×450

For 2D object detection, the benchmark uses SSD as a single-stage detector with a ResNet-50 backbone and multiple feature maps for multiscale detection. The associated MLCommons Cognata dataset contains 26 scenes, each 34 seconds at 30 FPS, with three forward-facing cameras and one rear camera plus LiDAR. The scenes span urban and highway settings under different light and weather conditions, and the dataset has about 36 objects per image on average. The benchmark uses camera images and 2D detection annotations; the exact class taxonomy is not specified in the paper.

For 2D semantic segmentation, the suite uses DeepLabv3+ with a ResNet-50 backbone on the same MLCommons Cognata dataset at 3840×2160. The label scope is semantic segmentation, but the exact class taxonomy is again not specified in the paper.

For 3D object detection, the benchmark uses BEVFormer-tiny with a ResNet-50 backbone. The model forms bird’s-eye-view representations from multiple cameras by means of spatiotemporal transformer attention. The dataset is nuScenes, which includes six cameras, five radars, one LiDAR, IMU, and GPS; in this benchmark, BEVFormer-tiny uses only the cameras. Each query consists of six camera images at 800×450. The nuScenes dataset is sampled at 12 FPS and contains 1,000 scenes totaling approximately 5.5 hours of driving, with annotations for 3D detection and tracking. The paper’s workload overview associates BEVFormer-tiny with target SAE level $\ge 3$ and approximately 45M parameters, SSD with target SAE level $< 3$ and approximately 14M parameters, and DeepLabv3+ with target SAE level $\le 3$ and approximately 40M parameters (Shojaei et al., 31 Oct 2025).

3. Reference models, portability, and engineering adaptations

All reference models are provided in ONNX format and are executed with ONNX Runtime through Python reference code. The reference implementations include the end-to-end pipeline, including setup, pre-processing, post-processing, LoadGen integration, and MLPerf-provided accuracy scripts. The stated purpose of the ONNX representation is to provide a static, platform-agnostic graph that submitters can translate into their own intermediate representations and compiler stacks (Shojaei et al., 31 Oct 2025).

The engineering burden of building portable references was nontrivial, particularly for BEVFormer. The benchmark team updated PyTorch from 1.7 to 1.13 and forked OpenMMLab dependencies to enable ONNX export; the final reference implementation runs with PyTorch 2.5 and ONNX Runtime. This centralized conversion effort removed a substantial barrier for submitters by turning research code into a deployable benchmark artifact.

The SSD workload required explicit training adjustments for the Cognata dataset and 8 MP inputs. The paper reports anchor box scale tuning to reflect the larger pixel footprint per object and the wide range of small or far versus large or near objects, expansion of the detection head kernel from $3\times3$ to $5\times5$, and the addition of a seventh feature map. With these changes, mAP improved from 0.6483 to 0.7141. Training SSD for 60 epochs on eight NVIDIA H100s took approximately 1.5 days. The post-processing stage includes non-maximum suppression, and the paper notes that NMS cost grows with scene complexity.

DeepLabv3+ is described as an encoder-decoder architecture with atrous separable convolutions, atrous spatial pyramid pooling, and skip connections, using a ResNet-50 backbone. The paper does not enumerate detailed normalization or other parameter values, stating instead that such specifics are defined by the reference implementation.

4. Metrics, formulas, and execution scenarios

MLPerf Automotive combines task accuracy metrics with latency metrics, but the dominant system-level performance criterion is 99.9% percentile tail latency. This is stricter than prior MLPerf inference suites and is justified as a mechanism for capturing jitter and rare delays that can compromise safety while remaining practical to measure within roughly a day rather than over much longer durations (Shojaei et al., 31 Oct 2025).

Accuracy is enforced as a fraction of the FP32 reference baseline. BEVFormer-tiny must retain at least 99% of FP32 accuracy, while SSD and DeepLabv3+ must retain at least 99.9%. The underlying task metrics follow standard detection and segmentation definitions, although exact IoU thresholds and class taxonomies are not specified in the paper and are delegated to the reference accuracy scripts. The benchmark states the following conceptual formulas:

$$\mathrm{IoU}(B_p,B_g)=\frac{|B_p \cap B_g|}{|B_p \cup B_g|}$$

$$\mathrm{AP}_c=\int_0^1 p_c(r)\,dr,\qquad \mathrm{mAP}=\frac{1}{C}\sum_{c=1}^{C}\mathrm{AP}_c$$

$$\mathrm{mIoU}=\frac{1}{C}\sum_{c=1}^{C}\frac{TP_c}{TP_c+FP_c+FN_c}$$

For latency, per-sample latency $L_i$ is measured by LoadGen, and the summary statistics include mean latency, percentile latency, and maximum latency:

$\bar{L}=\frac{1}{N}\sum_{i=1}^{N}L_i$

The benchmark reports latency under standardized scenarios rather than peak throughput numbers. In Single Stream, inherited from MLPerf Inference, LoadGen issues a new query immediately after the previous query completes, exercising peak inference rate at batch 1 with minimal queuing. In Constant Stream, LoadGen sends queries at a fixed rate and the system under test may be idle between queries. The fixed rates are 12 FPS for BEVFormer-tiny, aligned to nuScenes sampling, and 15 FPS for SSD and DeepLabv3+, chosen as an achievable target at 8 MP resolution. These scenarios are intended to represent, respectively, peak responsiveness under continuous load and periodic execution aligned to sensor sampling.

5. Evaluation protocol, divisions, and submission categories

MLPerf provides LoadGen, datasets, and accuracy scripts. The system under test loads samples, performs pre-processing and post-processing, executes inference, and returns results to LoadGen for latency and accuracy logging. Each workload requires two runs: a performance run to collect latency and an accuracy run to verify compliance with the required accuracy target. Compliance tests are required for each benchmark (Shojaei et al., 31 Oct 2025).

Closed division and open division encode different notions of comparability. Closed division prohibits retraining, caching results, and benchmark-aware preprocessing, and it enforces reproducibility and fair comparison across systems. Post-training quantization is allowed in closed division, subject to the required accuracy retention. Quantization-aware training is not permitted unless the QAT model is provided for all participants and accepted as a reference. Open division permits retraining, different models, and broader optimizations.

Submission categories further distinguish the maturity and safety posture of systems. A Hardened System meets functional safety requirements, is publicly available, and is auditable in closed division. A Development System does not meet full functional safety, is publicly available, and is auditable in closed division. An Engineering Sample is not functionally safe, is not publicly available, and is not auditable in closed division. In v0.5, power and energy measurement and thermal characterization are planned for future work and are not part of compliance; safety certification is represented through these categories rather than through formal certification in the benchmark round.

6. First-round submissions, limitations, and broader significance

The first round of v0.5 submissions comprised nine submissions from two organizations, with three submissions per benchmark workload. There was one Constant Stream submission, for BEVFormer-tiny, while all other submissions used Single Stream. Seven results were in the open division using MLCommons reference implementations and FP32 models, and two were in the closed division using vendor-optimized implementations. All submissions were in the Development System category. The software stacks included ONNX Runtime, PyTorch, and TensorRT; the systems ranged from an automotive compute platform to development servers; and the optimizations included INT8 PTQ, FP8, and FP16 reduced precision. The paper states that platform-specific performance and accuracy values are summarized on the MLCommons benchmark page but does not provide those numerical results in the text (Shojaei et al., 31 Oct 2025).

The benchmark paper also records the practical challenges encountered in constructing a public suite. Dataset acquisition required explicit permissions, nuScenes had to be approved for benchmark use, and synthetic Cognata data were procured to meet high-resolution and label-breadth requirements. SSD at 8 MP required substantial tuning to reach acceptable accuracy, and BEVFormer ONNX export required framework and dependency modernization. These details are significant because they show that benchmark standardization in automotive ML is conditioned not only by metric design but also by dataset governance and model portability.

Several limitations are explicit in v0.5. Power, energy, and thermal measurements are absent; safety-centric accuracy metrics for difficult or rare objects, zero-shot behavior, and temporal consistency are planned; submission categories are expected to be refined, including a relaxed functional safety category and a likely split of Development System into automotive versus non-automotive; and the workload scope is currently centered on perception rather than planning, prediction, digital cockpit, early sensor fusion, or end-to-end models including vision-language-action. A pre-silicon evaluation category is also planned. The public repository contains Python reference implementations with ONNX models, LoadGen integration, accuracy scripts, and scenario configuration, and is designed to run without accelerators while readily leveraging commodity AI accelerators (Shojaei et al., 31 Oct 2025).

In the broader automotive benchmarking landscape, MLPerf Automotive occupies a different role from platform-specific inference studies. A 2023 assessment of CNN packages on embedded automotive platforms evaluated YOLO and AlexNet on NVIDIA Jetson and Xilinx Zynq UltraScale+ systems using metrics such as mAP, FPS, end-to-end latency, board and SoC power, energy per frame, and performance-per-watt, but it explicitly noted that percentile latencies were not directly reported and that its power methodology would require alignment to MLPerf rules for official comparability (Burgio et al., 2023). This suggests that MLPerf Automotive’s main contribution is not merely the choice of automotive workloads, but the introduction of a standardized public methodology for tail-latency-constrained, accuracy-retention-qualified, and division-governed comparison of automotive ML systems.