Accelerating Precise End-to-End Simulation: Latency-Sensitive Many-core System Modeling

Abstract: Modern LLM workloads put increasing demands on parallel compute capability and on-chip memory capacity, while also stressing fine-grained data movement and synchronization. These trends motivate exploring and designing many-core accelerators with tightly coupled scratchpad memory (SPM) for scalable compute and predictable, explicitly managed data access. However, this architectural shift raises two challenges: cycle-accurate register-transfer level (RTL) simulation becomes prohibitively slow as system complexity grows, and performance estimation requires precise modeling of latency-sensitive interconnect behavior. This paper presents a fast yet accurate end-to-end modeling approach for latency-sensitive many-core architectures, targeting large-scale instances such as TeraNoC with 1024 cores and a 4MiB globally shared L1 SPM. The approach captures timing behavior of latency-sensitive SPM accesses across multiple interconnect scales, while abstracting non-essential hardware details. Across diverse benchmarks, the model tracks a cycle-accurate RTL golden model with errors below 7%, while delivering up to 115x faster simulation. The framework also provides detailed profiling across processing elements and interconnect, enabling efficient end-to-end software development and hardware design exploration. Two case studies demonstrate its practicality: profiling-guided optimization of FlashAttention-2 to reduce interconnect stalls and synchronization overhead, and design space exploration of network-on-chip (NoC) router remapping to alleviate traffic imbalance and improve throughput.

• The paper delivers a high-fidelity simulation framework achieving less than 7% error compared to RTL with up to 115x runtime acceleration.
• The methodology employs an event-driven approach to capture latency-sensitive data movements and synchronization bottlenecks in SPM-centric many-core architectures.
• Case studies on FlashAttention-2 and hardware DSE validate its practical impact on optimizing interconnect performance and resource allocation.

Accelerating Precise End-to-End Simulation: Latency-Sensitive Many-core System Modeling

Introduction

The paper addresses the growing requirement for accurate and efficient simulation of many-core architectures with tightly coupled software-managed scratchpad memory (SPM), a paradigm driven by contemporary LLM workloads. Such system configurations are increasingly prevalent due to their predictable memory access and scalable compute, but they also pose severe challenges in modeling and simulation, particularly regarding microarchitectural timing phenomena related to fine-grained, latency-sensitive data movement and synchronization. The work presents a framework implemented within GVSoC, targeting the TeraNoC architecture—a 1024-core instance with a hybrid mesh-crossbar network and a globally shared L1 SPM.

Problem Formulation and Significance

The complexity of SPM-centric many-core systems renders cycle-accurate Register-Transfer Level (RTL) simulation computationally infeasible for design-space exploration (DSE) and performance tuning when scaling to thousands of cores. Existing simulation approaches are either too abstract, thus missing critical timing fidelity, or too detailed, leading to prohibitive runtime. Critical system-level effects—such as contention-induced timing variations and synchronization bottlenecks in latency-sensitive, word-level request/response traces—are either inaccurately captured or entirely omitted by prior tools, especially in the presence of hierarchical interconnects with non-uniform traffic patterns.

Modeling Framework

The presented approach leverages the event-driven simulation paradigm of GVSoC, modeling the SPM interconnects using request-based abstractions and a parameterized router primitive. The model is explicitly designed to:

• Aggregate and propagate fine-grained timing metadata (including latency and serialized duration) at each interconnect hop, capturing both intrinsic and contention-induced delays.
• Focus on latency-sensitive phenomena relevant at the software-hardware boundary, abstracting away non-essential architectural detail for tractable simulation speed.
• Support both synchronous and asynchronous simulation strategies, selectively deployed to trade off fidelity and performance according to the criticality of the communication path (e.g., asynchronous for request paths in high-contention regions, synchronous elsewhere).
• Provide fine-grained cross-layer profiling, exposing execution traces, resource utilization, queueing, and conflict statistics for both software (kernels/PEs) and hardware (interconnect/memory).

Distinct interconnect regimes—within Tile, Group, and inter-Group—are systematically represented, enabling accurate modeling across the hybrid mesh-crossbar architecture. The abstraction also incorporates router remapping mechanisms for DSE studies on interconnect topology dynamism.

Validation and Results

The framework is validated against cycle-accurate RTL simulation using a suite of representative benchmarks spanning linear algebra, signal processing, and critical machine learning kernels like FlashAttention-2. The model demonstrates:

• End-to-end timing error below 7% across all benchmarks relative to RTL simulation.
• Simulation runtime acceleration of up to 115x over RTL methods, enabling rapid software/hardware design iteration.

Notably, these metrics are achieved while capturing detailed bottleneck phenomena, such as PE stalls due to non-uniform interconnect contention and synchronization imbalance, which cannot be reliably inferred from functional or packet-level simulation tools.

Case Studies and Implications

1. FlashAttention-2 Deployment and Optimization

Using the model's profiling data, execution breakdowns of FlashAttention-2 expose low PE utilization caused by interconnect-induced stalls and high synchronization overhead. Profiling reveals hotspots—specifically, port conflicts at remote Tile input ports. Guided by this insight, the authors optimize the kernel's data distribution and access schemes, reducing long-range NoC traversal and temporally smoothing critical accesses. The result is a significant reduction in interconnect stalls and synchronization delays, showcasing the model's value for software optimization targeting interconnect-sensitive computations.

2. Hardware DSE: NoC Router Remapping

A detailed DSE is performed using the model to examine traffic imbalance in statically mapped NoCs under global memory-intensive kernels such as MatMul. Link utilization and congestion analysis reveal underutilized capacity resulting from spatial and temporal imbalance. By introducing a dynamically configured crossbar switch ("router remapper") between Tiles and NoC routers, the framework demonstrates a throughput improvement of ≈10%, with optimized remapper partitioning yielding further gains (e.g., up to 47% for LayerNorm kernels). The ability to rapidly evaluate nuanced hardware modifications at scale is a direct consequence of the modeling approach's abstraction/performance trade-off.

Theoretical and Practical Implications

The methodology concretely advances the state-of-the-art in simulation of SPM-centric, many-core architectures. Its unified, word-level, timing-accurate abstraction bridges the gap between fully cycle-accurate but computationally infeasible simulation and lightweight, fast-but-coarse models that fail to accurately reflect microarchitectural bottlenecks. This enables:

• Iterative co-design/tuning of software kernels targeting tightly coupled SPM and hierarchical interconnects, with actionable profiling data.
• Rapid, fine-grained, and statistically meaningful architectural DSE, including evaluation of adaptive or dynamic interconnect schemes.
• Informed partitioning and resource allocation at both the software scheduling and hardware mapping layers for high-performance machine learning accelerators.

Future Directions

The demonstrated approach suggests further exploration in several directions:

• Extension to more complex and heterogeneous interconnect hierarchies—e.g., incorporating reconfigurability or adaptive topology management in response to workload patterns.
• Application of the modeling framework to system-level studies of emerging LLM accelerator architectures featuring advanced scratchpad, DMA engines, and hardware synchronization mechanisms.
• Integration with compiler/mapper toolchains for automatic guided optimization of interconnect- and memory-bound workloads based on detailed simulation feedback.

Conclusion

The paper delivers a high-fidelity, high-performance simulation methodology for SPM-centric, latency-sensitive many-core system modeling, implemented in GVSoC and demonstrated on TeraNoC. The framework achieves a robust trade-off between timing accuracy (within 7% of cycle-accurate RTL) and simulation speed (up to 115x faster), enabling rapid, profiling-driven design iteration for both hardware and software. The provided case studies evidence the framework's utility in both kernel-centric optimization and hardware DSE, establishing it as a valuable tool for research on next-generation many-core LLM accelerators and tightly coupled memory systems (2605.07750).