MegaScan: LLM Diagnostics

• MegaScan is a performance-diagnostics module in MegatronApp that uses low-overhead CUDA event tracing to monitor LLM training.
• It collects operator-level timing traces, reconstructs global cross-rank timelines, and aligns events by communication dependencies.
• The system leverages TP/PP/DP parallelism semantics to distinguish compute delays from communication bottlenecks, enabling targeted slow-node detection.

Searching arXiv for the relevant MegaScan paper and closely related work on distributed LLM training diagnostics. MegaScan is the performance-diagnostics module of MegatronApp, an open-source toolchain for distributed LLM training. It is a CUDA-event-driven tracing engine that collects operator-granular timing traces from every GPU rank in a Megatron-LM job, reconstructs a global, cross-rank timeline aligned by communication dependencies, and applies a multi-stage heuristic algorithm for online slow-node detection and root-cause localization (Zhao et al., 26 Jul 2025). Its design addresses stragglers and fail-slows in tightly coupled TP/PP/DP training, the lack of temporal causality in conventional monitoring, the lack of communication-pattern awareness in generic tracing, and the need to distinguish compute-dominated slowdowns from communication bottlenecks and ranks that are merely waiting. This emphasis on observability is continuous with large-scale training practice in systems such as MegaScale, which also relied on low-overhead CUDA event instrumentation, distributed traces, and topology-aware diagnosis to localize stragglers in production-scale LLM runs (Jiang et al., 2024).

1. Position in the MegatronApp stack

MegatronApp introduces four orthogonal, composable modules: MegaScan, MegaFBD, MegaDPP, and MegaScope. Within that stack, MegaScan is responsible for system bottleneck and straggler detection, whereas MegaFBD targets decoupled forward/backward placement, MegaDPP targets adaptive pipeline scheduling, and MegaScope targets interpretability and tensor-level visualization (Zhao et al., 26 Jul 2025).

MegaScan is explicitly specialized for Megatron-LM–style distributed training rather than generic cluster monitoring. It operates at operator level, is aware of TP/PP/DP group structure, uses CUDA events rather than heavyweight kernel profiling, and emits Chrome Tracing–compatible timelines for immediate visualization. Its core analytic question is: which rank, at which operator, under which parallelism dimension, is slowing the job, and why (Zhao et al., 26 Jul 2025).

Tool family	What it exposes	What MegaScan adds
DCGM and generic system monitors	GPU utilization, memory usage, system metrics per node	Cross-rank temporal causality and TP/PP/DP awareness
Traditional profilers	Kernel-level or iteration-level measurements	Low-overhead operator-granular tracing suitable for production
Generic distributed tracing	Generic RPC or system traces	Megatron structured collectives and P2P semantics

A plausible implication is that MegaScan should be read less as a general-purpose profiler than as a semantics-aware observability layer for tightly synchronized 3D-parallel training.

2. Trace acquisition and event representation

MegaScan is implemented as a lightweight tracing layer inside each Megatron-LM rank. It does not alter Megatron’s scheduling logic; instead, it injects CUDA events around GPU operations, records structured trace events, aggregates per-rank traces on rank 0, reconstructs dependencies, aligns clocks, and then runs analysis over the aggregated trace (Zhao et al., 26 Jul 2025).

For each operation, the tracer inserts a start CUDA event before the operation in the same stream and an end CUDA event after it. The elapsed time is obtained using cudaEventElapsedTime(start, end). Because the timestamps are taken in the GPU’s clock domain, the measured latency corresponds to true GPU latency rather than host-perceived latency. The system characterizes the instrumentation as low overhead and suitable for online use (Zhao et al., 26 Jul 2025).

Each event carries metadata in Chrome trace “Arguments,” including micro-batch index, communication payload size in bytes, peer rank or process group ID, and operation type such as GEMM, AllReduce, Send, or Recv. These fields are not incidental; they are the basis for dependency reconstruction, effective bandwidth computation, and cross-rank comparison (Zhao et al., 26 Jul 2025).

The fundamental per-operator quantity is

$$t_{\mathrm{op}} = t_{\mathrm{end}} - t_{\mathrm{start}}.$$

During training, each rank writes its own JSON trace. Rank 0 collects the per-rank files and persists aggregation in a separate thread so that the training path is not blocked. A post-processing step merges the traces into a single Chrome Tracing Format JSON file, with each rank represented as a separate process or thread track. The result can be loaded into chrome://tracing or Perfetto UI for visual inspection of start and end times, durations, overlaps, and idle gaps (Zhao et al., 26 Jul 2025).

3. Dependency reconstruction and global timeline alignment

The merged trace is not initially in a globally consistent time domain, because the timestamps originate from local GPU clocks. MegaScan therefore performs dependency reconstruction and timeline alignment before anomaly analysis (Zhao et al., 26 Jul 2025).

Dependency reconstruction identifies events that correspond to the same logical communication operation. For collectives such as AllReduce, AllGather, and Broadcast, the tracer records the global rank list of all participants. For P2P Send and Recv operations, it records the peer rank. A single pass over all events groups those with the same communication ID and participant set into one logical synchronous communication instance and attaches a cross-reference such as related_sync_op (Zhao et al., 26 Jul 2025).

Timeline alignment then uses these communication instances as anchors. A reference rank, for example rank 0, is selected. For each communication instance, MegaScan uses the start and end times on the reference rank as alignment anchors and adjusts other ranks for clock offsets. The key assumption is that for a synchronous collective or blocking P2P communication, all participants must logically finish that communication at the same global time before proceeding (Zhao et al., 26 Jul 2025).

Because Megatron TP/PP/DP training is communication-heavy, the system has many such anchor points. In TP there are frequent AllReduce and AllGather operations; in PP there are frequent Send and Recv calls between stages; in DP there is per-step gradient synchronization. The paper states that under heavy communication patterns, alignment becomes highly accurate and yields an almost perfect global timeline (Zhao et al., 26 Jul 2025). This is closely related to the role played by CUDA event traces and cross-rank timelines in MegaScale, where distributed event timelines were used to expose synchronization points, bubbles, and lagging ranks in large-scale production jobs (Jiang et al., 2024).

4. Slow-node detection and root-cause localization

MegaScan’s anomaly analysis is measurement-driven and heuristic-based rather than derived from a closed-form performance model. It exploits structural symmetries of Megatron-LM parallelism to identify slow operations, candidate slow ranks, and then root-cause ranks as distinct from collateral victims (Zhao et al., 26 Jul 2025).

The first stage is cross-DP comparison of compute kernels. In data parallelism, ranks with the same TP index and PP index but different DP index should execute identical compute sequences with similar latencies. For each logical operator type, such as a specific GEMM in a given layer, MegaScan compares execution times across those DP peers. When an instance exceeds a threshold relative to its peers, it is marked as a slow operation. For each rank $r$, MegaScan computes

$$p_r = \frac{\text{number of slow operations on rank } r}{\text{total operations on rank } r}.$$

Ranks whose $p_r$ exceeds a configured threshold are marked as candidate slow ranks (Zhao et al., 26 Jul 2025).

The second stage uses collectives to separate true sources from victims. Within a collective group, MegaScan compares collective start times across ranks. A true slow rank tends to start the collective later than its peers because its preceding computation is slower. If a rank is repeatedly the latest to enter collectives across many operations, it is classified as a likely root-cause slow node rather than a rank that is simply blocked by upstream delay (Zhao et al., 26 Jul 2025).

Pipeline parallelism requires different treatment, because staggered micro-batches make absolute start times non-comparable even in healthy executions. MegaScan therefore focuses on effective bandwidth for P2P transfers. For a P2P communication event $e$ with payload size $S_e$ and latency $L_e$,

$\mathrm{BW}_e = \frac{S_e}{L_e}.$

Statistics of $\mathrm{BW}_e$ are then computed over a window for each adjacent pipeline-stage pair. Persistently degraded bandwidth is treated as evidence of a true slow node, whether the cause is a NIC or PCIe issue or a stage that prepares or consumes data slowly. The analysis pays particular attention to the pipeline warm-up phase under 1F1B scheduling, where the forward-activation traffic pattern is comparatively clean (Zhao et al., 26 Jul 2025).

The outcome is a rank-level classification into root-cause slow ranks, collateral victims, and healthy ranks. This classification is precisely the capability that generic per-node telemetry lacks.

5. TP/PP/DP semantics and operational use

MegaScan is built around the semantics of Megatron-LM’s three principal parallelism dimensions. In DP, it uses replica symmetry: ranks with identical TP and PP coordinates should behave similarly on compute kernels. In TP, frequent synchronous collectives serve two purposes simultaneously: dense calibration points for timeline alignment and direct evidence for diagnosing late-arriving ranks inside a shard. In PP, where staggered scheduling invalidates naive time comparisons, the system relies on bandwidth-centric analysis of Send and Recv behavior (Zhao et al., 26 Jul 2025).

This division of labor across DP, TP, and PP is central to MegaScan’s design. It prevents communication-induced waits from being misread as GPU slowness and allows the analysis to distinguish hardware issues such as down-clocking, thermal throttling, and degraded links from software issues such as data skew or batch-size changes (Zhao et al., 26 Jul 2025).

Operationally, MegaScan is intended to remain enabled in production. The paper describes a workflow in which engineers turn on tracing with low code changes, allow each rank to log operator-level events and metadata to local JSON, inspect merged traces in Chrome Tracing or Perfetto when throughput regresses, and then run heuristic analysis to compute $p_r$, identify ranks consistently late to collectives, and flag anomalous P2P bandwidth in PP (Zhao et al., 26 Jul 2025).

The module is diagnostic rather than prescriptive. It does not automatically tune TP, PP, or DP degrees, nor does it rebalance micro-batch sizes. Instead, its outputs inform actions such as removing problematic GPUs, rebalancing a slow pipeline stage, or changing job placement so that critical TP or PP groups run on healthier or better-connected nodes. The surrounding MegatronApp modules, especially MegaFBD and MegaDPP, are positioned as mechanisms that could use this telemetry, although the paper does not detail explicit integration logic (Zhao et al., 26 Jul 2025).

6. Implementation boundaries, evaluation status, and future directions

MegaScan is built on CUDA events and integrated into PyTorch/Megatron-LM code paths, with Chrome Tracing Format JSON as the trace representation and Perfetto as the analysis backend. Its storage and querying workflow is straightforward: per-rank JSON traces are written during training, rank 0 gathers and merges them, and users then explore the merged trace either interactively in Chrome Tracing or Perfetto UI or programmatically through Perfetto’s SQL-like query engine (Zhao et al., 26 Jul 2025).

Perfetto SQL is not merely a convenience layer. It allows queries such as selecting all P2P transfers between two ranks and computing average bandwidth, or listing all collectives where a given rank started later than the group median. This makes the trace a queryable execution database rather than a static visualization artifact (Zhao et al., 26 Jul 2025).

The implementation is characterized as near-zero overhead and suitable for production, but no explicit numeric overhead measurement is reported. Likewise, the MegaScan paper does not provide quantitative evaluation tables or detailed case studies of the module in isolation. The strongest explicit effectiveness claim is qualitative: MegaScan enables practitioners to “isolate system bottlenecks in minutes rather than hours” (Zhao et al., 26 Jul 2025).

Its limitations are largely assumption-driven. The alignment and root-cause logic assume synchronous training and blocking collectives; cross-DP comparison assumes homogeneous logical roles for DP peers with the same TP and PP indices; the design is tailored to structured Megatron-style TP/PP/DP parallelism; the tracing is GPU-centric and does not directly profile CPU-side delay sources; and anomaly detection is heuristic and threshold-driven rather than backed by formal statistical guarantees. High-quality alignment also depends on communication-heavy workloads; communication-sparse jobs would provide fewer anchor points and therefore lower alignment precision (Zhao et al., 26 Jul 2025).

Future work centers on stronger reaction mechanisms after diagnosis. The broader MegatronApp roadmap mentions native support for fast failover after anomaly detection, which naturally suggests automated removal or migration of a slow rank, reconfiguration of parallel groups, or checkpointed restart after MegaScan identifies repeated anomalies. Additional tracing and diagnosis support for inference is also planned, reflecting the different latency and load constraints of serving workloads (Zhao et al., 26 Jul 2025).

In the broader literature of large-scale LLM systems, MegaScan occupies the observability and diagnosis layer. MegaScale demonstrated that low-overhead CUDA event monitoring, heat maps, distributed traces, and topology-aware visualization are indispensable once training spans thousands of GPUs and hidden stragglers dominate stability and MFU over time (Jiang et al., 2024). MegaScan systematizes that perspective into a dedicated module for Megatron-LM, combining operator-granular tracing, communication-aware alignment, and parallelism-specific root-cause analysis into a single diagnostic framework (Zhao et al., 26 Jul 2025).