MMHal-Bench Evaluation Framework

• MMHal-Bench Evaluation is an assessment framework that systematically characterizes and optimizes memory subsystem performance using modular design and polyhedral code transformations.
• It leverages configurable driver templates and scripted parameter sweeps to diagnose cache behavior, bandwidth ceilings, and synchronization overhead with reproducible results.
• Empirical findings demonstrate that fine-tuning thread synchronization, data layout, and multi-stream access significantly enhances performance metrics such as bandwidth and cache efficiency.

MMHal-Bench Evaluation is an assessment framework designed to systematically characterize and optimize application-specific memory subsystem performance. Drawing on techniques and architectural designs exemplified by AdaptMemBench, such evaluation separates driver infrastructure from access-pattern specification and leverages parameterized templates, polyhedral code transformations, and rigorous metric collection. The methodology extends to a broad class of memory-centric kernels, enabling reproducible, extensible measurement and fine-grained diagnosis of bandwidth, cache, and synchronization phenomena (Lakshminarasimhan et al., 2018).

1. Architectural Overview

Evaluation of MMHal-Bench employs a modular and highly configurable workflow to isolate variables relevant to memory subsystem performance. The architecture’s principal components are:

• Pattern specification: Headers (e.g., <kernel>.h) define memory allocation and mapping macros for kernel logic. Polyhedral input files (<kernel>_init.in, <kernel>_run.in, <kernel>_val.in) declaratively specify affine iteration domains and scheduling for initialization, main computation, and validation.
• Polyhedral code generator (optional): Tools such as ISCC/ISL map polyhedral descriptions into C code, supporting user-specified loop transformations (e.g., interchange, tiling).
• Driver templates: Two modes—Unified Data Spaces (shared arrays, OpenMP for-worksharing) and Independent Data Spaces (per-thread arrays, avoiding false sharing)—are provided, each with a PAPI measurement wrapper for hardware event capture.
• Automated build and execution: The framework combines generated kernel code and header logic into unified C++ drivers, compiled with aggressive optimization flags and OpenMP support. Parameter sweeps are automated over working set sizes, thread counts, and repetitions.
• Machine-readable output: For every run, wall-clock timing, PAPI counters, and metadata (including experiment configuration and applied code transformations) are recorded (Lakshminarasimhan et al., 2018).

2. Kernel-Independent Driver Templates

The driver template abstraction decouples boilerplate responsibilities—argument parsing, timing, setup of PAPI counters, and result serialization—from the computational kernel. This modularity facilitates rapid benchmarking of diverse access patterns. Typical loop structures are:

• Unified Data Spaces:

for(int k=0; k<ntimes; k++) { #pragma omp parallel for CLAUSE #include "<kernel>_run.c" }

• Independent Data Spaces:

#pragma omp parallel { int t_id = omp_get_thread_num(); for(int k=0; k<ntimes; k++) { #include "<kernel>_run.c" } }

Thread synchronization modes, scheduling, and chunking are made configurable through a CLAUSE macro. Switching templates or adjusting CLAUSE systematically explores thread-synchronization overhead, false sharing, and memory allocation strategies, all without modifying kernel code (Lakshminarasimhan et al., 2018).

3. Access Pattern Specification and Polyhedral Integration

Application-specific memory access is declaratively specified via kernel header macros (for allocation and statement mapping) and polyhedral inputs (defining loop domains and permissible transformations):

• Allocation and mapping, e.g.:

  #define Triad_alloc \ double *A=malloc(n*sizeof(double)); \ double *B=malloc(n*sizeof(double)); \ double *C=malloc(n*sizeof(double)); #define A_map(i) A[i] #define Triad_run(i) A_map(i)=B_map(i)+scalar*C_map(i)

• Polyhedral sets (_run.in) specify iteration domains in Presburger form and can be annotated for transformations:

  Domain_run := [n] → { STM_run[i] : 0 ≤ i < n };

The polyhedral engine (ISCC/ISL) enables sophisticated loop reordering, tiling, strip-mining, and can generate multiple loop nest variants from a single high-level specification. This accelerates exploration of optimizations and empirical studies of loop transformation impact on cache and bandwidth behavior (Lakshminarasimhan et al., 2018).

4. Performance Metrics and Measurement

MMHal-Bench evaluation reports comprehensive quantitative metrics:

• Wall-clock time: Aggregate kernel execution time across ntimes iterations.
• Memory bandwidth:

$$BW = \frac{V}{t} \quad\text{[bytes/sec]}, \quad V = n_{\rm elems} \times S_{\rm elem} \times n_{\rm streams} \times ntimes$$

• Achieved FLOP/s (for arithmetic kernels):

$GFLOP/s = \frac{F}{t \times 10^9}$

where $F$ is the total floating-point operation count.

• Cache and memory subsystem statistics: Using PAPI counters (e.g., L1_DCM: L1 data cache misses, CA_SHR: exclusive cache line requests).

This unified metric collection supports detailed apples-to-apples comparison between variants, elucidating cache locality limits, bandwidth ceilings, and effect of synchronization (Lakshminarasimhan et al., 2018).

5. Experimental Methodology and Workloads

Evaluations emphasize hardware-controlled repeatability and broad coverage:

• Experimental platform: Dual Intel Xeon E5-2680 v4 (2×NUMA, 14-core/socket), 32 KB L1, 256 KB L2 per core, 35 MB shared L3; cache line 64 B.
• Software stack: gcc 6.3 with -O3 -fopenmp, PAPI library for event counting.
• Benchmarks: STREAM-like triad (a[i]=b[i]+\alpha \cdot c[i]); 1D, 2D, and 3D Jacobi stencils.
• Variants: Unified vs. independent data spaces, array padding, interleaved/multi-stream access, full and partial tiling.

Experimental sweeps vary array size ($n$) to target different cache levels, thread counts, tile sizes, and number of simultaneous memory streams, with each configuration repeated and measured for stability (Lakshminarasimhan et al., 2018).

6. Empirical Findings: Case Study Results

The evaluation framework reveals the following through controlled design choices:

• OpenMP Barrier Overhead: Disabling implicit barriers (via nowait) increases L1–L2 bandwidth by up to 20%.
• Shared vs. Independent Spaces: Shared arrays suffer L1 contention, resulting in $BW_{L1} < BW_{L2}$; per-thread arrays mitigate cache-line sharing, approximately doubling L1 bandwidth.
• Multi-Stream Scaling: Increasing parallel memory streams from 3 up to 20 shows peak bandwidth at 11 streams. Interleaved data layouts (e.g., hexad, triad) enhance L1 bandwidth by up to 50% over naïve triads.
• False Sharing and Padding: Without padding, unified data spaces in Jacobi-1D evoke over 10× more L1 misses and high CA_SHR; padding arrays to cache line boundaries (stride 8) eliminates this effect and restores bandwidth.
• Multidimensional Stencil Tiling: Independent spaces improve raw bandwidth, but naive 2D/3D stencil tiling does not overcome L1 bottlenecks. Full 3D or partial (2D) blocking yields negligible reuse benefits.

A plausible implication is that memory subsystem performance for application-specific kernels is highly sensitive to granular details of data layout, loop scheduling, and synchronization policy (Lakshminarasimhan et al., 2018).

7. Guidelines and Lessons for MMHal-Bench Design

AdaptMemBench’s methodology informs MMHal-Bench evaluation best practices:

• Separation of driver and pattern: Maintain a clear boundary between driver logic and pattern/mapping to maximize modularity and reusability.
• Support both space modes: Implement both shared- and independent-space templates to diagnose code sensitivity to false sharing.
• Polyhedral frontend: Offer polyhedral code generation (ISCC/ISL) to automate systematic transformation of kernel loop nests.
• Unified measurement API: Collect time, bandwidth, cache miss and exclusive line request counters in a standard, inspectable format.
• Interleaving/multi-stream capability: Systematically probe memory system prefetch and OOO resources using multi-stream kernels, not just classical triad.
• Metadata-rich output: Embed experiment configuration and transformation history in machine-readable results for full reproducibility.
• Scripted parameter sweeps: Automate runs across tile sizes, data stream counts, thread counts to expose non-obvious architectural sweet spots (e.g., peak bandwidth at 11 streams).

By decoupling driver infrastructure, exploiting code generation for loop transformations, and embedding comprehensive metric collection and scripting, the AdaptMemBench methodology offers a robust and extensible evaluation template for MMHal-Bench and comparable memory-subsystem benchmarks (Lakshminarasimhan et al., 2018).