ACALSim: A Scalable Parallel Simulation Framework for High-Performance System Design Space Exploration

Abstract: Architectural simulation has become the critical bottleneck limiting design space exploration for high-performance computing systems. Modern GPUs and AI accelerators -- with hundreds to thousands of tightly-coupled components -- demand simulation frameworks that deliver efficient parallelism and scalable single-node execution. Existing frameworks fall short: SST focuses on multi-node MPI scalability but struggles with intra-node scaling, while GPGPU-Sim remains largely single-threaded. Critically, none expose a mechanism for users to optimize threading for their specific workloads. We introduce ACALSim, a scalable parallel simulation framework providing infrastructure and APIs for building high-performance simulators -- timing-model accuracy remains the responsibility of simulator developers. Its key innovation is a pluggable thread-management architecture that lets developers implement custom scheduling strategies tailored to specific simulation patterns, absent in existing frameworks. Complementing it are (1) event-driven execution with fast-forward to eliminate idle-cycle overhead, (2) a shared-memory data model enabling zero-copy communication, and (3) a two-phase parallel execution model for deterministic thread scaling. We demonstrate ACALSim through HPCSim, a GPU simulator targeting A100-class architectures. Against an SST implementation using identical shared timing cores to isolate framework overhead, ACALSim achieves over 14x speedup with 41% lower memory footprint; hardware validation confirms 0.72--1.22x cycle-count correlation with A100 measurements. While SST fails to complete 256+ thread-block workloads within practical time limits, ACALSim simulates full LLaMA transformer layers (single block) in 17.7 minutes for LLaMA-7B and 30.4 minutes for LLaMA-13B -- enabling design space exploration that SST cannot achieve.

• The paper introduces ACALSim, a simulation framework that enables scalable, cycle-accurate modeling with a novel two-phase parallel execution strategy.
• It integrates a pluggable ThreadManager and shared-memory abstractions to optimize heterogeneous simulator workloads and minimize overhead.
• Empirical evaluations demonstrate up to 14× speedup and lower memory usage compared to traditional frameworks, validating its efficiency for complex GPU and AI accelerator simulations.

ACALSim: A Scalable Parallel Simulation Framework for High-Performance System Design Space Exploration

Framework Architecture and Core Innovations

ACALSim targets a fundamental challenge in architectural simulation: the performance and scalability bottlenecks posed by tightly-coupled, cycle-accurate models necessary for high-performance GPU and AI accelerator DSE. The framework provides a deterministic two-phase parallel execution model (parallel component execution, followed by atomic commit) designed for reproducibility and scalability across hundreds to thousands of components within a single node. Its architectural abstractions feature event-driven execution with fast-forward (eliminating idle cycle overhead), shared-memory data exchange with zero-copy for efficient communication, and a pluggable ThreadManager interface that allows simulator developers to tailor scheduling for specific workload patterns.

Figure 1: ACALSim Framework Block Diagram illustrating SimTop coordination, SimBase entities, event-driven fast-forward, and modular thread/task management.

Figure 2: ACALSim Simulation Flow depicting deterministic two-phase iteration: parallel simulator dispatch and serialized control thread synchronization.

The development model prioritizes transparency: users implement simulator logic using single-threaded programming semantics, while ACALSim transparently handles parallelization, synchronization, and thread scheduling. The shared-memory architecture, through abstractions such as SimPort and SimChannel, ensures safe inter-simulator communication with automatically managed double-buffering and synchronization mechanisms.

Thread Management and Simulation-Pattern-Driven Optimization

A primary contribution of ACALSim is the user-extensible ThreadManager architecture, which distinguishes it from frameworks like SST, gem5, and GPGPU-Sim that enforce non-configurable threading models. This abstraction is central to handling divergent simulation patterns encountered in heterogeneous hardware modeling. Developers can select or implement custom thread scheduling strategies—such as thread pool mapping, priority ordering, lazy activation, or work-stealing—to maximize efficiency for sparse, memory-bound, or heterogeneously distributed simulation tasks.

Figure 3: Thread Manager Performance Comparison demonstrating scalable throughput via thread pooling as component count increases.

Integrated profiling tools analyze the available parallelism and identify overhead distribution, guiding the selection and tuning of ThreadManager variants. Empirical results show that simulation-pattern-driven thread management delivers up to 1.4$\times$ speedup (DGXSim: priority-ordered) and up to $>\!2\times$ improvement in scalability (GPUSim: thread-local queues), with lock contention elimination becoming critical at high thread counts.

Figure 4: TaskManager Scheduling Overhead Comparison highlighting reductions from specialized thread-local queue optimizations.

Case Study: Cycle-Accurate GPU Simulation

ACALSim's efficacy is validated by direct comparison against SST using HPCSim, an A100-class GPU simulator. Both simulators utilize identical timing cores, isolating framework overhead from modeling fidelity. ACALSim achieves over 14$\times$ speedup (GEMM 256$^3$, 64 TBs), with 41\% lower memory footprint (25.3 MB vs. 43 MB). SST fails to simulate larger workloads (above 256 thread blocks) within practical limits, timing out even on medium-scale configurations, whereas ACALSim completes LLaMA-7B and 13B single-layer inference (860/1,076 TBs) in 17.7 and 30.4 minutes, respectively.

Timing validation reveals 0.72–1.22$\times$ cycle count correlation with A100 hardware (∼20% error average), covering compute-bound, memory-bound, and bandwidth-bound scenarios. Fast-forward cycle elimination and parallel execution enable scalable throughput—ACALSim's advantages become increasingly pronounced as workload size and component coupling intensify.

Figure 5: Hybrid ACALSim-SST architecture for multi-GPU simulation, demonstrating composability of frameworks.

Figure 6: Multi-rank scalable deployment—2 SST ranks coordinating 4 GPUs—showing scale-out capabilities via hybrid integration.

Generalizability and Profiling-Driven Performance Enhancement

ACALSim has been adopted for diverse architectures (HPCSim: A100/H100 GPUs, DGXSim: 8-GPU systems, BlackBear: custom 64-PE accelerator), demonstrating applicability beyond GPU-centric models. Its modular interface supports cross-team integration, including C++/SystemC wrapping and hybrid SST orchestration for multi-device system simulations.

Profiling tools integrated within ACALSim facilitate in-depth analysis of simulation bottlenecks, available parallelism, and scalability limits. The profiling-driven approach enables targeted optimizations such as custom thread managers, workload-specific task ordering, and lock contention minimization. Empirical evaluation of DGXSim and GPUSim highlights memory-bound scaling limitations, sparse activation patterns, and overhead breakdowns—each addressed by simulation-pattern-specific ThreadManager design.

Figure 7: GPUSim Parallelism Degree Per Iteration illustrating the dynamic distribution of active simulator threads during execution.

Figure 8: Simulation Profiling Results providing execution time, phase breakdown, and scheduling overhead analysis.

Related Frameworks and Comparative Positioning

Existing frameworks such as SST, gem5, SystemC, ZSim, and ROSS feature varying strengths in distributed, modular, or optimistic parallel simulation. However, none provide user-extensible thread management or efficiently scale to cycle-accurate modeling of tightly-coupled accelerator architectures. SST's event-driven paradigm is optimal for loosely-coupled distributed systems but incurs prohibitive overhead for large-scale GPU workloads. ZSim's bound-weave synchronization is unsuitable given high communication/computation ratios in GPUs.

Recent AI-focused simulation tools (vTrain, ONNXim, LLMServingSim) emphasize application-specific profiling or fixed threading models, lacking the infrastructure flexibility and performance tuning capabilities of ACALSim. Critically, frameworks presupposing fixed threading policies are suboptimal for heterogeneous and unpredictable simulation workloads. ACALSim fills this gap via its pluggable ThreadManager API and simulation-pattern-driven optimization, providing necessary infrastructure for productive architecture research and DSE.

Practical and Theoretical Implications

Practically, ACALSim reduces the engineering effort to build cycle-accurate simulators, improves throughput by an order of magnitude, and enables architecture exploration for large-scale AI workloads previously rendered infeasible by simulation overhead. The deterministic execution model simplifies debugging, error injection, and timing verification, while enabling transparent scaling to hundreds-thousands of components.

Theoretically, ACALSim demonstrates that simulation-pattern-driven infrastructure customization—especially thread management—can fundamentally alter achievable performance ceilings in architectural simulation. The contrast with fixed-threading frameworks highlights the necessity for user-extensible abstractions, particularly as simulation scope expands from single-device to multi-device and system-level modeling. The hybrid ACALSim-SST integration suggests further opportunities for composable, unified simulation environments leveraging strengths of multiple paradigms.

Conclusion

ACALSim introduces a scalable, modular, and high-performance framework for cycle-accurate simulation of modern HPC and AI accelerator architectures. Its primary innovation—the pluggable ThreadManager—enables simulation-pattern-driven optimization, delivering robust performance improvements and memory efficiency unattainable by existing frameworks. Empirical case studies confirm strong accuracy and throughput on realistic workloads, and profiling-driven enhancements refine simulation scalability. As simulation complexity grows, ACALSim's infrastructure and extensibility position it as a fundamental enabler for productive, comprehensive design space exploration. Future work will extend thread management strategies, refine hybrid integration approaches, and address even larger-scale deployments.

(2605.22936)