Heterogeneity-Aware LLM Simulator

• The heterogeneity-aware distributed LLM simulator is a framework that models non-uniform hardware, memory, and interconnect interactions in both serving and training environments.
• It employs runtime-driven abstractions like execution planners, model serving groups, and heterogeneous device groups to capture scheduling, batching, and communication nuances.
• The simulator enhances predictive accuracy by integrating workflow-level metrics and synchronization costs under dynamic resource, memory, and network conditions.

Searching arXiv for the referenced simulator and scheduling papers to ground the article in current preprints. arXiv search query: "LLMServingSim 2.0 heterogeneous disaggregated LLM serving infrastructure" A heterogeneity-aware distributed LLM simulator is a system-level modeling framework for LLM execution on distributed infrastructures in which accelerators, memory systems, interconnects, and serving or training policies are not uniform. In recent work, such simulators are designed to make runtime-driven hardware–software interactions explicit and analyzable: batching, routing, offloading, memory movement, disaggregation, and communication are modeled inside a single execution loop rather than as isolated components. The topic spans both inference serving and distributed training, and it has become more prominent as LLM deployments shift toward heterogeneous and disaggregated clusters, online agentic workflows, and multi-tier memory and network hierarchies (Cho et al., 26 Feb 2026, Peng et al., 15 May 2026).

1. Definition and scope

The central problem addressed by heterogeneity-aware distributed LLM simulation is that homogeneous assumptions fail in practical deployments. On the serving side, modern infrastructures increasingly combine diverse accelerators and near-memory processing technologies, while system software separates computation, memory, and model components across distributed resources; performance therefore emerges from runtime interactions among scheduling, data movement, and interconnect behavior rather than from accelerator speed alone (Cho et al., 26 Feb 2026). On the training side, state-of-the-art simulators have often assumed homogeneous compute and network infrastructure, whereas practical environments contain mixed GPU generations, diverse interconnect technologies, and non-uniform communication paths (Kumar et al., 7 Aug 2025).

In this literature, heterogeneity is broader than “mixed GPUs.” LLMServingSim2.0 explicitly treats heterogeneous model types, devices, capacities, topologies, and serving structures as first-class configurable objects, and it describes heterogeneity as including system-level differences across serving topologies and policies (Cho et al., 10 Nov 2025). Training-oriented work adopts a similar expansion of scope: device groups, topology descriptions, device-to-parallelism mappings, and non-uniform workload allocation are all part of the simulator state rather than external assumptions (Kumar et al., 7 Aug 2025, Kumar et al., 25 Jun 2026).

A common abstraction for training is the heterogeneous device group. One paper defines a device group as

$$DG_1=\{(GPU_{type1}, count_1), \ldots, (GPU_{typeN}, count_N)\},$$

which allows a simulator to represent either homogeneous or heterogeneous collections of devices and to map them onto PP, TP, and DP roles (Kumar et al., 7 Aug 2025). This suggests that the simulator’s unit of reasoning is not necessarily a single accelerator; it may instead be a structured group with its own workload share, communication pattern, and synchronization role.

2. Architectural abstractions

Across the literature, heterogeneity-aware distributed LLM simulators converge on a small set of architectural abstractions. LLMServingSim 2.0 organizes serving around an Execution Planner, a Serving Engine containing one or more Model Serving Groups (MSGs), and a System Simulator. At runtime, a Request Router dispatches requests to MSGs, a batch scheduler forms executable batches, a memory model checks fit and cache state, an operation mapper assigns operators to devices, an operation scheduler builds an execution DAG with communication and memory-movement steps, and the System Simulator executes that graph at cluster scale (Cho et al., 26 Feb 2026).

A related serving-oriented formulation models the system as a set of heterogeneous instances coordinated by a global request router. In that design, an instance encapsulates its own scheduler and memory model; additional first-class abstractions include prefill instances, decode instances, an expert router for MoE, and a radix-tree-based prefix cache manager (Cho et al., 10 Nov 2025). The same paper emphasizes that routing decisions are affected by resource status, load balancing, workload characteristics, and cache state, which means that dispatch cannot be reduced to a static placement rule.

Training simulators expose an analogous but not identical structure. Xsim is presented as a full-system, discrete-event simulator that covers workload generation, non-uniform model/data/pipeline partitioning, collective communication, pipeline execution and synchronization, and network behavior (Kumar et al., 25 Jun 2026). H2, while introduced as a training framework rather than a standalone simulator, provides a simulator-relevant stack consisting of DiTorch, DiComm, HeteroPP, HeteroAuto, and topology-aware resharding with fine-grained overlap (Tang et al., 23 May 2025).

System	Primary domain	Salient abstractions
HexAGenT (Peng et al., 15 May 2026)	Agentic serving target	Online-revealed DAGs, prefill/decode placement, KV transfer, workflow SLOs
LLMServingSim2.0 (Cho et al., 10 Nov 2025)	Serving simulator	Instances, global request router, P/D disaggregation, MoE, prefix cache manager
LLMServingSim 2.0 (Cho et al., 26 Feb 2026)	Unified serving simulator	Execution Planner, MSG, operation mapper, System Simulator
Xsim (Kumar et al., 25 Jun 2026)	Training simulator	Device groups, pipeline barriers, multi-ring communication, NS-3/htsim backends
SimAI extension (Kumar et al., 7 Aug 2025)	Training simulator design	Custom device groups, non-uniform partitioning, heterogeneous network modeling

Taken together, these abstractions indicate that a heterogeneity-aware simulator is typically runtime-driven, queue-aware, and graph-based. Requests or operators are not merely assigned to devices; they are mapped into evolving execution graphs that include memory transfers, synchronization, and topology-specific communication.

3. Serving-side simulation and workflow awareness

Serving-oriented simulators have moved beyond single-request latency models toward systems that represent distributed request routing, prefill/decode disaggregation, MoE execution, prefix caching, and cache-aware scheduling. LLMServingSim2.0 supports multiple heterogeneous instances, global request routing, P/D disaggregation, expert routing and offloading for MoE, and prefix caching with network and memory awareness; requests may be forwarded together with KV cache after prefill, and the cache-transfer policy is configurable (Cho et al., 10 Nov 2025). The unified LLMServingSim 2.0 formulation further embeds these serving decisions into a runtime loop that explicitly couples batching, routing, offloading, memory hierarchy, interconnects, and power (Cho et al., 26 Feb 2026).

HexAGenT is not itself a simulator, but it formalizes a serving regime that a heterogeneity-aware distributed LLM simulator must reproduce if it is to evaluate modern agentic workloads. In HexAGenT, each request is an online-revealed DAG: at arrival time only source nodes are known, child calls appear after parent completion or tool return, and scheduling is event-driven on workflow arrival, prefill completion, transfer completion, and decode completion. The cluster is a heterogeneous prefill-decode disaggregated system with mixed A100, H100, and H200 GPUs, explicit KV transfer between stages, and distinct compute, memory, and transfer constraints for prefill and decode (Peng et al., 15 May 2026).

The paper’s workflow-level urgency model illustrates the shift from per-call to end-to-end objectives. For workflow $w$, the revealed subgraph $G_w(t)$ has an isolated completion estimate $H_w(t)$, and a ready call $c$ is scored by

$R_s(c,t) = \frac{(t-a_w)+\Delta_s(c,t)}{H_w(t)},$

where $a_w$ is workflow arrival time and $\Delta_s(c,t)$ is the projected elapsed time until completion at stage $s \in \{\mathrm{Prefill}, \mathrm{Decode}\}$ (Peng et al., 15 May 2026). The corresponding SLO objective minimizes the scale factor $\alpha$ needed to attain a target completion fraction:

$w$0

with $w$1 (Peng et al., 15 May 2026). Req95 and Req99 are then the minimum scale factors required for 95% and 99% timely completion.

This workflow-aware formulation corrects a common misconception: in agentic serving, optimizing individual LLM-call latency is not equivalent to optimizing user-perceived latency. HexAGenT reports that across representative agentic workloads and heterogeneous A100/H100/H200 clusters, the required SLO scale is reduced by an average of 20.1% at 95% attainment and 33.0% at 99% attainment, with maximum reductions of 45.0% and 80.5%, respectively (Peng et al., 15 May 2026). A plausible implication is that any serving simulator that omits online DAG revelation, KV-transfer latency, or decode-capacity feasibility can reproduce per-call traces while still mischaracterizing workflow-level SLO attainment.

Argus extends the serving perspective to heterogeneous edge-cloud deployment. It models clients, $w$2 edge servers, $w$3 cloud servers, time-slotted arrivals, server-specific compute capacity $w$4, communication rates $w$5, per-device queues $w$6, and token-length-aware workload estimates. Its Length-Aware Semantics module predicts output token length, and its scheduling layer combines prefilling, decoding, communication delay, and queueing into offloading decisions under long-term constraints (Wu et al., 28 Dec 2025). This suggests that token-length prediction can itself be a first-class simulator component rather than a preprocessing artifact.

4. Training-side simulation and non-uniform execution

Training-oriented heterogeneity-aware simulators reject the assumption of equal partitioning. The SimAI-based extension for heterogeneous compute and network infrastructure identifies non-uniform workload partitioning, resharding for shape mismatch, heterogeneity-aware collective communication, and heterogeneous compute and interconnect simulation as required components (Kumar et al., 7 Aug 2025). In this line of work, different device groups may receive different numbers of layers, different tensor-parallel degrees, or different batch portions depending on compute and communication capability.

Xsim develops this approach into a full-system simulator for distributed LLM training on heterogeneous AI clusters. It supports non-uniform partitioning of layers, micro-batches, TP degrees, and pipeline-stage depths across device groups, and it exposes metrics such as pipeline bubble time and straggler waiting time (Kumar et al., 25 Jun 2026). For DP synchronization under heterogeneous TP layouts, Xsim constructs customized communication rings using

$w$7

where $w$8 is the TP degree of device group $w$9, and then applies LCM-based chunking so that each participating rank contributes the same amount to each chunk (Kumar et al., 25 Jun 2026). The paper also introduces a sweep-line method for forming DP synchronization groups from overlapping layer ranges, with complexity

$G_w(t)$0

where $G_w(t)$1 is the number of device groups and $G_w(t)$2 is the number of unique layer segments (Kumar et al., 25 Jun 2026).

A complementary direction models heterogeneous nodes together with dynamic network topology changes. The automatic parallelization framework built around simulation-based search represents the network as a dynamic temporal graph with multi-edge physical link abstractions and uses branch-and-bound to search over operator placement, splitting, fusion, and communication scheduling. Feasibility is determined by dependency, memory, and bandwidth constraints, including

$G_w(t)$3

for link bandwidth at time $G_w(t)$4 (Wu et al., 3 Jun 2025). This is a training-side analogue of serving simulators’ queue and cache feasibility checks.

addresses an even more heterogeneous regime: a hyper-heterogeneous cluster over 1,000 chips. Its simulator-relevant contribution is a cost model and search space for heterogeneous pipeline parallelism. The framework searches over per-chip-type pipeline stages, TP degree, layer allocation, recomputation, and DP degree under the constraint

$G_w(t)$5

and evaluates heterogeneous training throughput with

$G_w(t)$6

(Tang et al., 23 May 2025). In reported experiments on a 100-billion-parameter LLM, H2 achieves up to 16.37% improvement over baseline homogeneous training solutions, while search overhead is reported as 0.62 s, 5.48 s, and 12.29 s in three experiments (Tang et al., 23 May 2025).

Across these systems, non-uniformity is the defining property. Faster devices are not simply given more replicas; they may receive different operator fragments, different pipeline stages, different tensor shapes, or different synchronization responsibilities.

5. Modeling methodology, fidelity, and metrics

A major methodological divide in this literature is between cycle-accurate integration and profile- or trace-driven system simulation. LLMServingSim2.0 explicitly adopts trace-driven performance modeling with an operator-level profiler implemented in PyTorch, using hooks between LLM layers to measure layer-wise latency and collect traces that drive the simulator. The paper argues that this allows new accelerators to be integrated with a single-line command and reports 232× faster execution on average for profiling/integration relative to repeatedly doing cycle-accurate hardware simulation; in its TPU-v6e-1 case study, integration required 258 LoC rather than 4764 LoC, an 18.5× reduction (Cho et al., 10 Nov 2025).

The unified LLMServingSim 2.0 paper extends profile-based modeling to latency and power, describing per-operator hardware profiles as inputs to a runtime loop implemented on a modified ASTRA-sim and Chakra. It reports an average error of 0.97% across validated settings while keeping simulation times around 10 minutes even for complex configurations (Cho et al., 26 Feb 2026). Xsim adopts a different fidelity tradeoff by supporting both NS-3 and htsim: NS-3 provides packet-level simulation, whereas htsim provides faster flow-level simulation and is reported as 16×–44× faster than NS-3 in some settings and up to 47× faster in some network configurations (Kumar et al., 25 Jun 2026).

Validation metrics also differ by domain. Serving simulators report TTFT, TPOT, ITL, throughput, memory usage, prefix-cache hit rate, power consumption, total energy consumption, and workflow-level scaled-SLO measures such as Req95 and Req99 (Cho et al., 26 Feb 2026, Peng et al., 15 May 2026). Training simulators emphasize predicted training time, Flow Completion Time for collectives, pipeline bubble time, straggler waiting time, and TCO; Xsim defines

$G_w(t)$7

as an exposed metric for deployment planning (Kumar et al., 25 Jun 2026).

The numerical results reported in the literature are strong but method-dependent. LLMServingSim2.0 reports 1.9% error for reproducing GPU-based serving, with error within 5% across evaluated configurations and simulation of 100 requests in under 12 minutes (Cho et al., 10 Nov 2025). Xsim reports less than 5% error across most heterogeneous DP/TP configurations and around 2% error for pipeline-parallel communication modeling, while noting that SimAI can mispredict heterogeneous deployments by up to 80% in a motivating example (Kumar et al., 25 Jun 2026). These figures should not be treated as interchangeable, because the modeled workloads, system boundaries, and fidelity targets differ.

6. Limitations, misconceptions, and research directions

Several misconceptions recur in discussions of heterogeneity-aware distributed LLM simulation. One is that heterogeneity is primarily a hardware taxonomy problem. The surveyed work instead treats heterogeneity as runtime interaction among hardware, memory hierarchy, network topology, routing, offloading, queueing, cache state, and parallelism structure (Cho et al., 10 Nov 2025, Cho et al., 26 Feb 2026). Another misconception is that distributed LLM simulation can be reduced to average latency prediction; workflow-serving papers show that end-to-end objectives such as Req95 and Req99 depend on online dependency revelation, stage-coupled placement, and KV-cache feasibility, none of which are captured by a homogeneous single-call abstraction (Peng et al., 15 May 2026).

The literature also states clear limitations. LLMServingSim2.0 notes that accuracy is harder in multi-instance settings because of request routing variability and network contention, that P/D disaggregation is more error-prone than dense multi-instance serving because instances must coordinate and exchange intermediate results, and that MoE is harder to reproduce accurately because gating changes token routing across layers and batches; future work mentions extending to CXL devices (Cho et al., 10 Nov 2025). Xsim assumes GPU homogeneity within a node, models GPipe-style execution while still extending support for 1F1B and other schedules, and uses htsim as an approximation rather than a packet-level replacement (Kumar et al., 25 Jun 2026). H2 relies on accurate deployment specifications and profiled operator behavior, and its contribution is primarily a training framework with simulator-relevant cost models rather than a standalone general-purpose simulator (Tang et al., 23 May 2025).

Taken together, the recent literature indicates two converging directions. The first is unified, runtime-driven serving simulation in which routing, batching, caching, offloading, and power are simulated together on heterogeneous and disaggregated infrastructures (Cho et al., 26 Feb 2026). The second is non-uniform training simulation in which device groups, resharding, dynamic networks, and pipeline imbalance are made explicit (Kumar et al., 7 Aug 2025, Kumar et al., 25 Jun 2026). A plausible implication is that future heterogeneity-aware distributed LLM simulators will be judged less by whether they contain a particular hardware model and more by whether they preserve the coupled runtime phenomena that determine latency, throughput, memory behavior, and synchronization cost in real deployments.