DeepFlow-based Frontend: Hardware-Aware LLM Simulation

• DeepFlow-based Frontend is an analytical framework that transforms high-level LLM and hardware specifications into detailed per-GPU execution traces with hardware-aware cost estimates.
• It employs a three-stage pipeline that maps workloads, models operator-level latency using tile-based estimates, and linearizes compute and communication events for simulation.
• The framework enables exhaustive exploration of hybrid-parallelism and sharding configurations, predicting execution time and memory usage with high fidelity for LLM training and inference.

A DeepFlow-based frontend refers to an analytical performance modeling framework component designed to transform a high-level specification of LLM workloads and hardware into detailed, hardware-aware execution traces. In the context of RAPID-LLM, the DeepFlow-based frontend enables operator-level simulation of LLM training and inference under complex hybrid-parallelism strategies and advanced sharding policies, while factoring in detailed hardware characteristics and memory constraints (Karfakis et al., 22 Dec 2025).

1. Architectural Overview

The DeepFlow-based frontend in RAPID-LLM is built to interface abstract LLM descriptions and explicit hardware specifications, producing per-GPU Chakra execution traces suitable for downstream network and congestion simulations. Its two principal responsibilities are:

• Ingesting LLM workload and hardware configurations, including hybrid-parallelism (data, tensor, pipeline, sequence, and context parallelism, plus ZeRO/FDSP sharding).
• Annotating the resulting operator-level compute graph with hardware-aware cost estimates (latency, memory footprint), followed by linearization of this graph into ordered event traces for “Chakra”—the execution model used.

Inputs include model shape (layer count $L$, $d_{\mathrm{model}}$, attention heads, vocabulary, prefill/decoding lengths), parallelism configuration (DP/TP/PP/SP/CP), sharding stage or placement policy (ZeRO-1/2/3, FDSP), and per-GPU architectural specs (SM count, peak throughput, cache/bandwidth, HBM).

The pipeline consists of three stages:

1. Workload mapping and operator-graph synthesis according to selected parallelism axes.
2. Per-operator cost modeling using tile search, resource occupancy, and multi-level memory estimation.
3. Generation of per-GPU Chakra traces interleaving compute and communications by dependency order.

2. Abstract LLM Specification to Hardware-Aware Chakra Trace

DeepFlow operates via a staged process to instantiate simulation-executable operator traces:

1. Hybrid Parallel Partitioning: Given degrees of data, tensor, pipeline, sequence, and context parallelism, operators (e.g., attention blocks, MLPs) and microbatches are mapped to GPU ranks. TP and SP axes partition tensor shapes for GEMMs or activations; DP and PP axes shape dependencies.
2. Operator and Tile Candidate Extraction: For each operator (e.g., QKᵗ MatMul, FlashAttention, softmax reduction), a handful of tile strategies $\{\tau_1, \tau_2, ...\}$ are enumerated, each defining tile size for threadblocks.
3. Latency Estimation per Tile: Each candidate tile is evaluated for:
   • Streaming Multiprocessor (SM) occupancy and “waves” required,
   • SRAM/L2/HBM movement volumes from tile geometry,
   • Multi-level roofline-based latency calculation:

   $$T_{\mathrm{tile}}(\tau) = \max\left( \frac{\mathrm{FLOPs}_\tau}{R_\mathrm{comp}\cdot \eta_\mathrm{occ} / w}, \frac{B_{\mathrm{SRAM},\tau}}{BW_{\mathrm{SRAM}}}, \frac{B_{\mathrm{L2},\tau}}{BW_{\mathrm{L2}}}, \frac{B_{\mathrm{HBM},\tau}}{BW_{\mathrm{HBM}}} \right)$$

   Where $w$ is waves, $\eta_\mathrm{occ}$ is occupancy.

The tile with minimum predicted latency is selected for the operator.

1. Operator Graph Construction: Each operator node is labeled with latency and memory footprint attributes; edges denote data dependencies and collective sync points (AllReduce, AllGather, etc.).

2. Trace Linearization: For each GPU rank, a topological walk emits a sequence of compute and communication events annotated with computed latency or transferred bytes, including explicit dependency ordering for correct simulation.

3. Tile-Based Latency Modeling

DeepFlow’s latency model is a tile-centric, analytical roofline estimator parameterized by kernel tiling, hardware occupancy, and hierarchical memory usage.

• SM Utilization and Wave Count:

Let $\mathrm{SM}_{\max}$ be SMs per GPU, $T_{\max}$ max threads per SM, $\mathrm{TB}_\tau$ threads per block, $N_{\mathrm{TB}}$ total blocks:

$$\eta_\mathrm{occ} = \min\left(1, \frac{\mathrm{SM}_{\max} \cdot T_{\max}}{N_{\mathrm{TB}}\cdot \mathrm{TB}_\tau}\right)$$

$$w = \left\lceil \frac{N_{\mathrm{TB}}}{\mathrm{SM}_{\max}} \right\rceil$$

• Memory System Traffic:

  • $B_{\mathrm{SRAM},\tau}$: within-tile bytes transferred (register/shared memory).
  • $$B_{\mathrm{L2},\tau} = \text{(tile data)} \times \mathrm{L2\,miss\,rate}$$, with
  • $$\mathrm{miss\_rate}_{L2} \approx \min\left(1, \frac{\mathrm{working\_set}_\tau}{C_{L2}}\right)$$.
  • $$B_{\mathrm{HBM},\tau} = B_{\mathrm{L2},\tau} \times \mathrm{miss\_rate}_{L2}$$.
• Latency Synthesis:

The tile latency is the bottleneck among four quantities: flops per occupancy per wave (compute-bound), SRAM, L2, or HBM bandwidth. Each operator selects the minimal-latency tile over admissible candidates for subsequent trace annotation.

4. Activation-Liveness Traversal and Memory Pruning

To guarantee feasible per-GPU memory consumption under complex parallelism and sharding, the DeepFlow frontend tracks live activation usage and static state over the execution trace. The process:

• Tracks a “StoreSet” comprising parameter, optimizer state, gradients, KV-caches, and static buffers, adjusted for ZeRO sharding (ZeRO-1: optimizer, ZeRO-2: gradients, ZeRO-3: parameters) and FDSP shifts.
• For every graph event, the set of live activations is updated:
  • “ComputeForward” events allocate activation tensors.
  • “Backward” events, or recompute policies, cause selective deallocation/recomputation and release of used activations.
  • Peak memory usage is tracked against physical capacity.
• Any parallelism or recomputation scheme leading to infeasible peak usage (exceeding GPU RAM) is pruned from further analysis.

A key portion of the logic is defined by a linear scan over the event trace, allocating and freeing tensors as dictated by graph dependencies and the current recompute policy (full or selective checkpointing).

5. Analytical Assumptions, Trade-Offs, and Model Limitations

Design decisions in DeepFlow emphasize analytical tractability and rapid parallelism exploration at the cost of some simulation detail:

• Analytical Modeling: DeepFlow does not execute or simulate actual kernels or packet flows but uses closed-form occupancy, roofline, and reuse-distance approximations to predict latency and memory usage.
• Tile-Search Practicality: Only a small candidate set of per-operator tilings is evaluated, trading off exhaustive search against practical speed; sub-optimal tiling may introduce minor latency variance (usually a few percent).
• Communication Abstraction: Collective operators are represented as abstract Chakra events, with backend expansion adding network topology and congestion modeling.
• Hierarchical Execution Modes: At cluster scales, the frontend may partition runs per-layer and microbatch (“hierarchical mode”) to reduce trace size, speeding exploration but reducing cross-stage overlap fidelity.
• Calibration Options: Correction factors (for network overlap or memory derating, e.g., due to thermal throttling) can be optionally fit from measurements and included as hardware spec knobs.

This suggests a primary trade-off in DeepFlow-based frontends is between analytical model fidelity and exploration throughput; the choice of coarse-grained tiling and abstracted collectives accelerates design space search but omits certain fine-grained execution details. This architectural stance directly facilitates exhaustive evaluation of hybrid-parallel and sharding configurations in the RAPID-LLM ecosystem (Karfakis et al., 22 Dec 2025).

6. Context, Use Cases, and Extensibility

The DeepFlow-based frontend enables fast, scalable, and relatively accurate what-if analysis of LLM training and inference strategies on complex GPU clusters. It predicts execution time and memory feasibility for hybrid-parallel LLM workloads, providing operator-level traces for simulation backends that incorporate congestion, routing, and hardware faults (e.g., HBM bandwidth throttling, degraded links). Validation on A100-based workloads demonstrates high predictive fidelity: Llama inference latency and GPT-scale training step times within 10.4% of published measurements, and communication simulation within 8% of ns-3 packet-based models.

Case studies include exhaustive sweeps over hybrid-parallel configuration space, sensitivity quantification to soft link faults, and rapid evaluation of hypothetical future GPU designs. The modularity of per-operator modeling, explicit parallelism mapping, and memory pruning heuristics allow the DeepFlow-based frontend to scale with future hardware features and evolving LLM architectures (Karfakis et al., 22 Dec 2025).