Sim-FA: A GPGPU Simulator Framework for Fine-Grained FlashAttention Pipeline Analysis

Abstract: To efficiently support LLMs, modern GPGPU architectures have introduced new features and programming paradigms, such as warp specialization. These features enable temporal overlap between the producer and consumer, as well as between matrix multiplication and activation function operations, substantially improving performance. To conduct effective AI infrastructure and computer architecture research, cycle-accurate simulators that support these new features, together with analytical models that faithfully capture workload characteristics, are essential. However, existing academic tools provide limited support for these emerging requirements. Existing cycle-accurate simulators do not incorporate new NVIDIA GPU features, such as the Tensor Memory Accelerator (TMA), in a timely manner. Moreover, existing analytical models can misestimate DRAM traffic under certain configurations. In this paper, we build a simulation pipeline from FlashAttention-3 kernel instrumentation to cycle-accurate simulation. The simulator achieves a mean absolute percentage error (MAPE) of 5.7\% and a maximum absolute percentage error of 12.7\% against H800. We also provide a theoretical analysis of FlashAttention-3 and explain why existing analytical models can produce inaccurate traffic estimates.

• The paper introduces a cycle-accurate simulation framework that models asynchronous, warp-specialized FlashAttention-3 kernels, achieving a MAPE of 5.7% in latency predictions.
• The framework accurately captures key GPU primitives like TMA, mbarrier, and WGMMA to resolve bandwidth bottlenecks, cache contention, and pipeline stalls.
• Empirical validation on Hopper (H800) and Blackwell (GB10) silicon shows robust predictive accuracy and informs optimization strategies for GPU resource allocation.

Sim-FA: A Cycle-Accurate GPGPU Simulation Framework for FlashAttention Pipeline Analysis

Introduction

The proliferation of LLMs has catalyzed a significant shift in GPU architecture and programming paradigms, with new features such as NVIDIA Hopper’s TMA (Tensor Memory Accelerator) and WGMMA (WarpGroup MMA) forming the bedrock for high-throughput, low-latency transformer operator implementations. However, the lack of academic infrastructure supporting these features has hindered principled systems and architectural research. "Sim-FA: A GPGPU Simulator Framework for Fine-Grained FlashAttention Pipeline Analysis" (2605.00555) addresses this gap by introducing a trace-driven, cycle-accurate simulation framework designed to model the asynchronous, warp-specialized execution of FlashAttention-3 kernels. This work delivers detailed modeling at the granularity of modern GPU primitives, resolving the deficiencies of prevailing tools that fail to accurately capture timing, bandwidth bottlenecks, and asynchrony, while also offering an analytical traffic model validated on both Hopper (H800) and Blackwell (GB10) silicon.

Figure 1: The Sim-FA framework enables trace-driven, cycle-level modeling of FlashAttention-3 pipelines, faithfully capturing asynchronous execution and hardware-software co-design complexity.

Modeling Asynchronous GPU Execution

Sim-FA defines a architectural abstraction where the WarpGroup rather than the thread or threadblock becomes the unit of scheduling and synchronization, reflecting the reality of TMA/WGMMA-driven pipelines. Its core simulation pipeline parses instrumented FlashAttention-3 kernel traces, deduplicates events, and reconstructs all resource dependencies, including sophisticated memory and compute overlays.

Key features of the Hopper architecture supported by Sim-FA include:

• TMA: Hardware-accelerated address generation and asynchronous HBM↔shared-memory transfers, with explicit modeling of descriptor caches, setup overhead, inflight limits, and L2 behaviors.
• mbarrier: Hardware barriers with fine-grained semantics, enabling precise modeling of pipeline bubbles and divergent code scheduling.
• WGMMA: WarpGroup-level asynchronous Tensor Core launches with explicit commit/wait semantics, modeled with per-shape calibrated latencies.

This design facilitates accurate modeling of tightly interleaved producer-consumer pipelines, cache contention, and pipeline stalls erupting from resource contention at the granularity observed on real hardware.

Microarchitectural Calibration and Bandwidth Analysis

The model incorporates microbenchmark-driven calibration for all major timing-relevant resources. TMA setup latency is resolved into fixed and variable components matched to Hopper hardware [luo2025hopper]. A critical contribution is Sim-FA’s modeling of the partitioned L2, employing a behavioral RemoteCopy proxy to mirror measured latency regimes between LLC hit, cross-partition, and DRAM-bound working sets.

Figure 2: TMA request path in Sim-FA, discriminating bulk versus tensor requests and accurately modeling setup, descriptor caching, and inflight tracking.

Figure 3: H800 TMA latency validation. Sim-FA closely matches empirical measurements across L2-hit, transition, and DRAM-plateau regions.

Bandwidth sensitivity to finite MSHR pool sizing is explored, yielding an empirically justified setting (MSHR=256) that aligns simulation and hardware, with minimal (5.79%) MAPE across challenging, bursty TensorMap traffic scenarios.

Figure 4: TMA bandwidth is highly sensitive to the number of modeled L2 MSHR entries; Sim-FA is tuned to minimize global error.

Figure 5: Bandwidth validation across 1D, 2D, and 3D TensorMap copies, capturing bottlenecks unique to high-dimensional asynchrony.

End-to-End FlashAttention-3 Simulation

Sim-FA demonstrates robust predictive accuracy for FlashAttention-3 kernel latency, achieving a MAPE of 5.7% (worst-case 12.7%) across Llama-3 model sizes and sequence length regimes. Simulated and measured Gantt charts elucidate the explicit overlap of producer and consumer WarpGroups as well as intra-/inter-group pipelining that is central to modern high-utilization kernel design.

Figure 6: End-to-end latency prediction for FA3 kernel; simulated and measured performance closely track across Llama-3 workloads.

Figure 7: Gantt chart for simulated FA3 pipeline: producer and two consumer WarpGroups exhibit designed overlap and synchronized tile access.

Ablation studies underscore the timing and performance significance of:

• Accurate L2 request coalescing (LRC), which is necessary to avoid bandwidth inflation from duplicate requests.
• Realistic hash-based multi-slice L2 allocation to prevent pathological slice contention under regular access patterns.
• Aggressive TMA address deduplication, without which simulated bandwidth explodes and error exceeds 500%.

Analytical Modeling and Empirical Validation

Sim-FA is complemented by SimFA-python, an analytical modeling framework distinct in its concurrency-aware, cache/bandwidth-limited traffic estimation. Unlike prior analytical models (e.g., GenZ), it delineates ideal and realistic DRAM traffic regimes, switching when $K/V$ working sets exceed effective L2 capacity.

Figure 8: SimFA-python validates L2 cache traffic predictions against hardware for various Llama-3 workloads on GB10.

Validation demonstrates that for long-sequence, long-context LLM inference, idealized models systematically underpredict off-chip traffic, missing refetch-induced scaling regimes that are workload- and architecture-dependent. SimFA-python accurately captures the phase transition from cache-contained to DRAM-rebounded memory behavior.

Figure 9: DRAM traffic: SimFA-python tracks measured traffic and reveals the breakdown of ideal cache assumptions. Baselines (GenZ) frequently under- or over-estimate actual traffic depending on the working set regime.

Implications and Future Directions

The necessity of a full-featured, warp-specialized, asynchronous simulator is now acute for architectural and systems research targeting the latest LLM accelerator designs. By resolving resource bottlenecks at the level of TMA, L2 partitioning, and pipeline bubbles, Sim-FA enables realistic cycle-level exploration of pipeline schedules, resource allocation, and memory hierarchy for multi-modal, long-context workloads that dominate emerging deployment scenarios.

The findings indicate new optimization vectors for both compiler and hardware architects. L2 cache bottlenecks and resource allocation policies (e.g., MSHR sizing, request coalescation) are now primary determinants of achievable throughput. The concurrency-aware analytical methodology is essential for forward-looking co-design, and naive scaling using traditional analytical models will mislead both architecture and performance studies. Further, as operator and hardware scaling trends persist, models suggest operator behavior will become SRAM-bound, motivating additional research into tiling, cache management, and bandwidth scaling.

Conclusion

Sim-FA constitutes the first academic, cycle-accurate simulator for emerging warp-specialized LLM kernels, capturing the complexity and performance characteristics of Hopper/Blackwell-era GPUs. The framework achieves strong empirical agreement (5.7% MAPE) with real silicon for FlashAttention-3, validates an analytical traffic model that corrects and supersedes prior frameworks, and establishes new methodology for systems research and hardware co-design. These advances make Sim-FA an indispensable tool for evaluating the next generation of LLM-centric accelerators and scheduling abstractions (2605.00555).