DMDBench: MD & VQA Benchmark

• DMDBench is a dual-resource benchmark comprising a proxy-app for molecular dynamics kernels and a curated validation set for LVLM visual question answering.
• The MD-Bench component employs a maintainable C99 framework with configurable data layouts, neighbor lists, and SIMD-optimized kernels to analyze short-range interactions.
• The VQA DMDBench provides 1,000 real-world, annotated images of digital devices to assess LVLM performance under challenging scenarios like occlusion and low-light.

DMDBench refers to two distinct but independently-developed resources in computational research: (1) a proxy-application for high-performance benchmarking and deconstruction of short-range molecular dynamics (MD) kernels (Machado et al., 2023); and (2) a curated real-world image validation set for visual question answering (VQA) on digital measurement devices, intended for benchmarking large vision-LLMs (LVLMs) in challenging scenarios (Valente et al., 29 Aug 2025). Each is designed as a community benchmark with rigorous protocols, quantitative metrics, and detailed ground-truth annotations. Both are intended to enable systematic performance evaluation and to stimulate methodological innovation by providing well-understood standards and measurement regimes.

1. Scope and Motivation

In computational science, reproducible benchmarks are essential for measuring progress and diagnosing bottlenecks. The first DMDBench, developed at the Erlangen National High Performance Computing Center, was designed as a maintainable “sandbox” in C99 for in-core analysis of MD kernels, offering reference implementations of force evaluation and neighbor list construction, and exposing detailed performance levers uncluttered by full-application overhead (Machado et al., 2023). Its focus is on short-range potentials implemented in production MD packages such as LAMMPS and GROMACS, with the goal of supporting research in high-performance computing and code optimization.

In a distinct domain, the DMDBench dataset introduced with the CAD2DMD-SET tool was curated to address robustness failures in LVLMs when reading values from digital measurement devices (DMDs) in real-world contexts. This DMDBench serves as an evaluation set for assessing VQA accuracy under conditions of clutter, occlusion, poor lighting, extreme viewing angles, and motion blur, with all images and annotations made explicitly structured for downstream analysis (Valente et al., 29 Aug 2025). Its driving motivation is to enable the community to quantify and improve LVLM performance on practical, error-prone VQA scenarios.

2. MD-Bench: Design, Architectures, and Core Algorithms

MD-Bench provides a transparent, extensible proxy-app that captures the most computationally expensive operations of short-range MD simulations while eschewing complex domain-specific infrastructure. Its architecture is modular, composed of C99 libraries and optionally hand-written SIMD/assembly kernels. The top-level workflow reads atomic coordinates from standard formats, initializes a cell-list or neighbor-list structure, and executes force calculations using a variety of algorithmic cores (Machado et al., 2023).

The primary algorithms are:

• Verlet List and Linked-Cell Construction: Both full neighbor lists (FN, every pair counted) and half lists (HN, exploiting Newton’s third law: $F_{ij} = -F_{ji}$), supporting both scalar and vectorized/clustered force evaluation.
• Force Kernels: Implemented for both Lennard-Jones (LJ) and Embedded Atom Model (EAM) potentials, each exists in scalar, software-gather, and hardware-gather variants.
• GROMACS M×N Supercluster Kernels: Organize atoms into clusters of $M$ and $N$ for SIMD-optimized inner and outer loops, with architecture-specific unrolling for AVX-512.

Performance measurement is enabled by integration with LIKWID, supporting cycle counting, CPI measurement, and bandwidth quantification. The build system is highly configurable, facilitating rapid switching of compilers, SIMD target flags, data layout schemes (AoS, SoA, AoSoA), and arithmetic precision.

Table 1: Key MD-Bench Design Features

Component	Options / Approaches	Purpose
Data Layout	AoS, SoA, AoSoA	Load alignment and gather cost
Neighbor List	Full (FN), Half (HN)	Exploit symmetry, reduce ops
Kernels	Scalar, SW-gather, HW-gather, Hand-tuned SIMD	Performance and architecture eval
Potentials	Lennard-Jones, EAM	Generality
Instrumentation	LIKWID markers, static analysis tools	In-core bottleneck diagnosis

MD-Bench supports “stubbed” tests (fixed neighbor indices/patterns) and “gather-bench” microbenchmarks to isolate specific microarchitectural effects, such as gather latency and control-flow divergence.

3. MD-Bench: Usage Studies and Performance Engineering Insights

Extensive studies conducted with MD-Bench decompose the effects of vectorization, compiler quality, memory access patterns, and floating-point precision (Machado et al., 2023):

• Floating-Point Precision: Single-precision yields significant speed-ups only in highly vectorized, cluster-layout kernels (GROMACS M×N); for gather-bound kernels (LAMMPS Verlet list), SP reduces runtime by only ~15%. Use of hardware reciprocal plus Newton–Raphson refinement leads to negligible thermodynamic error while improving runtime by ≈10%.
• Assembly Code Analysis: Compilers differ radically in loop unrolling, handling of SIMD remainders, and vectorization completeness. ICC can emit specialized prologs for partial vectors, while others default to scalar fallback.
• Latency and Divergence: Memory latency overhead is low (<12%) in representative use-cases; branch/control-flow divergence incurs modest penalties (4–5%), which can be dissected with stubbed workloads.
• Code-Quality Impacts: HN lists outperform FN in unvectorized (scalar) settings, but with increasing SIMD width FN becomes more favorable due to greater vectorization efficiency and better masking utilization.
• Compiler Comparison: ICC is generally fastest and most complete in SIMD utilization; Clang 15 narrows the gap considerably, while GCC lags in reducing scalar ops and fails to shrink instruction count for SP.

Main insights:

• GROMACS-style M×N clustering is more SIMD-friendly than classical gather-dependent approaches.
• Hardware-gather instructions have high latency and can bottleneck wide-vector kernels unless data layout is optimized for streaming.
• Both architectural and compiler advances are required for optimal vector utilization, mask handling, and epilogue minimization.

4. DMDBench for LVLM VQA: Dataset Composition and Annotation Protocol

The DMDBench dataset from CAD2DMD-SET comprises 1,000 real-world images of digital measurement devices, annotated for fine-grained VQA under practical and adversarial capture conditions (Valente et al., 29 Aug 2025). It is strictly a validation/test set; no train split is provided.

Dataset structure:

• Collection Context: Images were acquired with smartphones and AR headsets across laboratory, workshop, and hospital environments. Devices are positioned with diverse backgrounds, occlusions, and lighting scenarios.
• Device Types: Multimeter, bench power supply, metronome, digital thermometer, pulse oximeter, blood-pressure monitor.
• Variation: ~35% near-frontal, ~40% oblique, ~25% extreme angles; 20% low-light, 30% strong glare; 15% occlusion; 25% with motion blur; 50% heavy clutter.
• Annotation Protocol:
  • Each image is annotated with one or more VQA question–answer (“Q–A”) pairs targeting key readings (numeric, units, mode).
  • A region of interest (ROI) bounding box delineates the display area, specified as (x, y, w, h) in JSON.
  • For multiple display fields, each sub-field is a sub-ROI with discrete Q–A pair.
  • Alternative valid answers per field are enumerated to accommodate format variation.

Table 2: DMDBench Q–A Annotation Example

Image file	Device	ROI(s)	Q	G (Ground-truth)
lab_bench_035.jpg	Multimeter	[430,120,300,120]	DC voltage reading	["12.3 V", "12.30 V"]
ward_oximeter_210.png	Pulse Oximeter	[120,340,180,80]; [350,340,120,80]	SpO₂ value; pulse rate	["97 %"]; ["72 BPM", "72 bpm"]
workshop_psu_088.jpg	Power Supply	[250,220,260,100]; [550,220,200,100]	Output voltage; Output current	["13.8 V"]; ["1.20 A", "1.2 A"]

5. Evaluation Protocol, Metric, and Benchmark Results

The standard DMDBench benchmarking protocol for LVLM VQA follows these steps (Valente et al., 29 Aug 2025):

1. Fine-tune LVLM or LoRA adapter on the synthetic CAD2DMD-SET training set.
2. Freeze model parameters.
3. Evaluate on all 1,000 DMDBench images, issuing the appropriate VQA prompt per field.
4. Calculate the Average Normalised Levenshtein Similarity (ANLS) per Q–A:

$$ANLS = \frac{1}{N}\sum_{i=1}^N \max_{g\in G_i}\left[1 - \frac{d_{\mathrm{lev}}(\hat y_i,g)}{\max(|\hat y_i|,|g|)}\right]_+$$

where $N$ is the number of Q–A pairs, $d_{\mathrm{lev}}$ is the Levenshtein distance, $G_i$ is the set of acceptable gold answers, and $[\cdot]_+$ clips negatives to zero.

Performance metrics are reported as overall ANLS, as well as breakdowns for word-level, unit, and numeric accuracy. Device-type and scenario-specific scores are also reported.

Table 3: Selected Benchmark Results (InternVL2.5, DMDBench @50% fine-tuned)

Scenario	ANLS (%)	Word Acc. (%)	Unit Acc. (%)	Numeric Acc. (%)
Multimeter	98	—	—	—
Power Supply	96	—	—	—
Metronome	94	—	—	—
Thermometer	95	—	—	—
Pulse oximeter	97	—	—	—
Blood pressure	93	—	—	—
Motion-blur (all devices)	94	—	—	—
Low-light	92	—	—	—
Extreme-angle	90	—	—	—
Heavy clutter	95	—	—	—

InternVL2.5-26B, following fine-tuning with a 50% synthetic (CAD2DMD-SET) mix, achieved up to 96% overall ANLS and improved numeric accuracy from 51% (base) to 69%.

6. Limitations, Extensions, and Practical Recommendations

In the MD domain, MD-Bench does not currently implement long-range electrostatics or distributed (MPI) parallelism, though it does provide realistic test cases for metals and supports further extension to emerging hardware (e.g., GPU-based supercluster kernels, auto-tuning frameworks) (Machado et al., 2023). It is not intended as an end-user simulation code, but as a low-noise performance analysis micro-benchmark.

For DMDBench VQA, the current test set is limited to six digital device types, comprised entirely of indoor images; there is no coverage of analog gauges, oscilloscopes, or outdoor sunlight scenarios (Valente et al., 29 Aug 2025). Anticipated dataset expansions include incorporation of analog dial meters, temporal consistency (sequence) tasks, explicit error-mode detection (e.g. "overload", "OL" readings), and synthetic augmentation via wear-and-tear on CAD models.

Recommended practices include:

• Validate LVLMs on DMDBench after any fine-tuning with synthetic data to avoid overfitting.
• Use separate one-word labels for numeric and units for optimal OCR-style performance; this has improved unit accuracy from 84.5% to 99.6% in InternVL2.5.
• A LoRA adapter trained on just 10% synthetic data can yield >90% ANLS.

This suggests that DMDBench is positioned as a rigorous, community-standard performance and accuracy validation suite in both molecular simulation algorithmics (Machado et al., 2023) and multimodal machine learning for VQA (Valente et al., 29 Aug 2025).

7. Broader Significance and Outlook

DMDBench exemplifies modern benchmark design: transparent, narrowly focused on the “hottest” computation or perception cases, and accompanied by open, interpretable ground truth and metrics. In the MD case, this enables systematic co-design and optimization at the interface of hardware, compiler, and software microarchitecture, and exposes fine-grained bottlenecks for exascale readiness. In the LVLM scenario, it offers a challenging real-world testbed for evaluating multimodal VQA systems on critical, commercially relevant tasks, setting a standard for measuring robustness and generalization.

Expected future extensions across both domains include integration with advanced auto-tuning and meta-benchmarking frameworks, additional device and measurement modalities, and an expanded emphasis on interpretability of errors (disentangling numeric, unit, and domain-specific failure modes). With the trajectory of both hardware and model scale, such focused benchmarks as DMDBench remain essential for rigorous, reproducible progress.