Extended Abstract: Re-Evaluating the Real-System Modeling Accuracy of Ramulator 2.0

Abstract: Cycle-level DRAM simulators provide accurate and flexible models for DRAM and memory controller operations and enable research on current and future memory systems. Therefore, they are critical for improving the performance, efficiency, and robustness of DRAM-based memory systems. Ramulator 2.0 (successor of Ramulator) is a highly modular and extensible cycle-accurate DRAM simulator that enables rapid exploration of new ideas in DRAM-based memory systems. A MICRO 2024 best paper runner-up publication, A Mess of Memory System Benchmarking, Simulation and Application Profiling, which we refer to as "the Mess paper," with all three artifact badges awarded (including "Reproducible"), proposes a new benchmark to evaluate real and simulated memory system performance. While doing so, it makes strong negative claims about Ramulator 2.0 and shows unexpected results. In this talk and the associated extended abstract, we demonstrate that these results and claims in the Mess paper are incorrect and are due to configuration and simulator usage errors made in the Mess paper. We describe four best practices to aid users and developers of simulation tools to avoid such issues in the future. We emphasize the importance of contacting simulator authors and developers when unexpected results are observed (especially and importantly before publishing such results), to ensure these simulators are used with correct configurations and as intended. Our investigation also aims to stimulate discussion on artifact evaluation practices and on mechanisms for correcting results and artifacts after publication. To aid future works and reproduction of all our results, we open source all our code and scripts at https://github.com/CMU-SAFARI/Cleaning-up-the-Mess. We refer the reader to our full ISPASS 2026 paper and its artifact for the complete analysis, detailed methodology, and extended results.

• The paper demonstrates that properly configured Ramulator 2.0 achieves cycle-accurate DRAM modeling closely matching real-system measurements.
• It identifies critical methodological and configuration flaws in previous evaluations that led to significant underestimations of DRAM bandwidth and latency.
• It underscores best practices in simulation, emphasizing adherence to JEDEC standards, transparent configurations, and robust artifact disclosure.

Re-Evaluating Ramulator 2.0: Correcting Misconceptions in Real-System DRAM Modeling

Introduction

The paper "Extended Abstract: Re-Evaluating the Real-System Modeling Accuracy of Ramulator 2.0" (2606.14566) scrutinizes the claims made by Esmaili-Dokht et al. in "A Mess of Memory System Benchmarking, Simulation and Application Profiling" (the Mess paper), which criticized the cycle-level DRAM simulator Ramulator 2.0 for significant modeling inaccuracies relative to real-system measurements and attributed these discrepancies to fundamental simulation errors. This extended abstract identifies major methodological and configuration flaws in the Mess paper’s evaluation, systematically demonstrates the accurate modeling capability of Ramulator 2.0 when used with correct configurations, and advocates best practices for simulator usage and benchmarking in memory systems research.

DRAM Simulation and Benchmarking Context

Ramulator 2.0 is a highly modular cycle-level DRAM simulator designed for extensible modeling of modern and next-generation memory systems. Its accuracy and flexibility have rendered it a central evaluation tool for DRAM performance, efficiency, and robustness studies. The Mess paper introduced a synthetic benchmarking methodology intended to characterize memory bandwidth and latency profiles, but reported results from Ramulator 2.0 that were sharply negative: simulated latencies were described as "unrealistically low," and peak bandwidth was less than half of the real-system measurements (126 GB/s simulated vs. 292 GB/s measured). These claims shaped community perception and led to artifact evaluation awards, despite subsequent discovery of substantial configuration and methodological errors.

Root Causes of Inaccuracy: Analysis and Correction

A detailed audit of the Mess paper’s open-source artifacts, scripts, and trace files revealed critical errors:

Channel Width and Memory System Configuration:

The Mess paper wrongly assumed that a Ramulator DDR5 channel was 64 bits wide (based on two logical 32-bit sub-channels) and extrapolated single-channel simulated bandwidth by a factor of 8 instead of 16. In reality, each DDR5 channel is 32 bits wide per the JEDEC DDR5 standard, meaning their extrapolation compared an 8-channel simulation to a 16-channel real-system configuration, drastically underestimating achievable bandwidth.

Unrealistic Simulation Latency Parameters:

Mess’s evaluation scripts configured Ramulator with zero cache latency and an abnormally large count of Miss Status Holding Registers (MSHRs), leading to artificially short round-trip memory latency and suppressed queueing delays. Such parameters are not representative of real-world CPU-memory interaction and invalidate latency results.

Methodological Opaqueness and Reproducibility Failures:

Key configurations and methodological divergences were neither disclosed nor justified, with source modifications hardcoding non-realistic values. Moreover, necessary artifacts to reproduce Ramulator results were unavailable for over ten months after artifact evaluation badges were awarded, undermining the reproducibility criterion.

Corrected Evaluation: Ramulator 2.0 with Proper Configuration

The authors developed a new frontend for Ramulator 2.0—the Mess Request Generator—precisely emulating the Mess benchmark’s workload characteristics (mixed stream/pointer-chase access patterns, controllable NOP intervals, variable read/write ratios). Evaluating a properly-configured, 16-channel (32-bit each) DDR5-4800AN system with realistic parameters and sweeping memory intensity variables yielded bandwidth-latency profiles closely matching real-system results.

Figure 1: Comparative bandwidth-latency profiles in Mess benchmarking for (a) Ramulator 2.0 as simulated in the Mess paper, (b) the real ARM system, and (c) corrected Ramulator 2.0 with proper configuration.

Key findings include:

• Maximum Achieved Bandwidth: Correctly configured Ramulator achieves a maximum bandwidth of 281.1 GB/s, precisely converging with analytically calculated limits (281.2 GB/s) accounting for periodic all-bank refresh delays.
• Bandwidth-Latency Curvature: Increased write ratios produce sharper latency-bandwidth curves due to DRAM bus turnaround penalties, concordant with real-system phenomena.
• Latency Scaling: Random read latency escalates with bandwidth utilization, reflecting queueing interference from stream accesses, as expected in hardware queues.

The corrected results directly contradict the Mess paper’s assertion of "poor resemblance" between Ramulator simulation and real-system performance, affirming Ramulator 2.0’s cycle-accurate modeling when used as intended.

Implications and Best Practices

This re-evaluation foregrounds critical procedural and scholarly standards for simulation-based research:

• Simulator users must consult JEDEC and tool documentation, not rely on colloquial technical assumptions—misinterpretation of standards can undermine validity.
• Configurations must reflect real architecture parameters; all modifications and methodology differences must be disclosed for scientific scrutiny.
• Open communication with simulator maintainers is imperative when unexpected results arise; the open-source community model demands peer engagement for quality control.
• Rigorous artifact review and disclosure must precede publication, especially when making strong claims about freely available simulators.

Adoption of these best practices is essential to prevent propagation of erroneous science, safeguard the integrity of benchmarking infrastructure, and support reproducibility across the computer architecture community.

Practical and Theoretical Implications

Practically, Ramulator 2.0 remains a sound infrastructure for DRAM research. The clarified methodology facilitates accurate evaluation of emerging DRAM standards, channel configurations, and system-level optimizations. Theoretical implications extend to improved modeling reliability for memory-centric computing, in-memory processing, and large-scale accelerator evaluation.

Future developments should emphasize composable simulator paradigms (e.g., Ramulator 2.1 [luo2026ramulator2.1]), automated artifact validation, standardized benchmark suites, and an ecosystem of reproducible tools supporting next-generation DRAM technologies. Artifact evaluation processes must include verification of methodological disclosure and direct consultation with tool authors, especially for open-source artifacts.

Conclusion

The extended abstract authoritatively invalidates the Mess paper’s claims regarding Ramulator 2.0, attributing previous inaccuracies to demonstrably incorrect configuration and usage. When applied rigorously, Ramulator 2.0 achieves cycle-accurate DRAM modeling that aligns closely with real-system results. The analysis underscores the necessity for thorough documentation, open communication, and methodological rigor in simulation studies. These lessons are broadly applicable to future research in memory system evaluation, simulator artifact publication protocols, and education of architecture researchers. The continued evolution and open-source dissemination of DRAM simulators like Ramulator remain central to the reliability and progress of memory system research.