CCL-Bench: Trace-Based LLM Evaluation

• CCL-Bench is an evidence-centric benchmarking framework that uses execution traces, YAML workload cards, and launch scripts to document LLM workload executions.
• It packages comprehensive artifacts to diagnose system performance across hardware accelerators, interconnects, communication libraries, and parallelism strategies.
• The benchmark shifts focus from scalar metrics to post-hoc analyses and simulation-backed what-if studies, enabling deeper insights into performance bottlenecks.

CCL-Bench 1.0 is a trace-based benchmark for LLM infrastructure that evaluates hardware, software, and parallelization choices through reusable execution evidence rather than only end-to-end summary numbers. Its central premise is that claims such as “workload $X$ is fastest on hardware $Y$ with software $Z$” depend on a large configuration space spanning accelerators, interconnect bandwidth, software frameworks, parallelism plans, and communication libraries, and that conventional scalar benchmark outputs do not explain why one configuration outperforms another. Each benchmark entry therefore packages an execution trace, a YAML workload card, and the launch scripts, allowing fine-grained analysis of compute, memory, and communication behavior and enabling post-hoc metric computation on previously collected runs (Ding et al., 7 May 2026).

1. Definition and benchmarking rationale

CCL-Bench was proposed to address a specific limitation of existing LLM infrastructure evaluation: most benchmark suites publish a small set of end-to-end metrics, but those metrics are insufficient for diagnosing bottlenecks, attributing performance differences, or asking new questions after the run has completed. In the benchmark’s formulation, infrastructure performance depends jointly on hardware accelerators, interconnect topology and bandwidth, framework and compiler choice, communication library, and the full parallelism plan, including DP, TP, PP, EP, and related settings. CCL-Bench therefore treats benchmarking as the collection of reusable evidence for each workload execution rather than as the publication of a fixed scoreboard (Ding et al., 7 May 2026).

This design positions the benchmark between several existing traditions. Relative to end-to-end suites such as MLPerf, LLM-Perf, and vendor leaderboards, it emphasizes explanation over scalar ranking. Relative to communication microbenchmarks and library-specific reports, it preserves full workload context rather than isolating a single primitive. Relative to trace-analysis tools such as Holistic Trace Analysis, it adds standardized workload cards, a public contribution model, a shared trace pool, and versioned metric tools. Relative to simulators and analytical models such as Astra-Sim, SimAI, Vidur, LLMCompass, and vTrain, it grounds performance analysis in measured traces rather than only calibrated abstractions. This suggests that CCL-Bench is best understood as an evidence-backed repository of workload executions and the tooling needed to interpret them (Ding et al., 7 May 2026).

2. Evidence model, artifact structure, and submission workflow

A CCL-Bench submission consists of three required artifacts: an execution trace, a YAML workload card, and the launch scripts. The execution trace is the primary evidence artifact. Accepted formats include Kineto / PyTorch profiler JSON traces for GPU and related accelerator backends, JAX/XLA profiler traces via XProf for TPU/JAX workloads, and optionally Nsight Systems traces when hardware-specific counters are required. These traces record operators, kernels, communication events, timestamps, streams, and per-rank activity, so that both coarse metrics such as step time and finer analyses כגון kernel breakdowns, collective overlap, memory-transfer overhead, and communication traffic can be computed (Ding et al., 7 May 2026).

The workload card makes the trace interpretable. Representative fields include provenance; workload and model metadata such as phase, whether the model is MoE, model family, precision, iteration count, parameter count, embedding parameter count, layer count, heads, and head dimension; workload shape such as batch size and sequence length; hardware information such as accelerator model, XPU count, topology, and network bandwidth; software information such as framework, compiler, parallelism plan, communication library, and communication environment variables; and metric-source fields describing which traces and auxiliary artifacts are available. The launch scripts preserve the actual launch commands, environment definitions, communication-library settings, framework options, and tuning choices, which the benchmark treats as part of the evidence because tuning effort is otherwise invisible in conventional benchmark reports (Ding et al., 7 May 2026).

The submission workflow is correspondingly structured. A contributor runs a workload, collects traces, attaches the workload card and scripts, and submits the entry. CCL-Bench then performs rule-based checks for schema validity, environment completeness, and minimum iteration count, after which the toolkit computes metrics and the entry can be published after review. For training submissions, contributors must attach launch scripts and collect at least 5 warm-up iterations followed by at least 5 steady-state iterations. For inference, the serving engine is warmed up to complete CUDA graph capture and KV-cache initialization, and traces must cover the prefill pass and at least 128 steady-state decode steps. The benchmark reports a profiling overhead of 0.22% on average over 6 steps, with coefficient of variation 2.67%, supporting routine trace collection. Raw trace access is gated on contributing a trace for the same workload, while workload cards, scripts, and tools are public (Ding et al., 7 May 2026).

The benchmark also formalizes analysis as a function from workload card and trace to a scalar metric,

$$f_i : \mathcal{W} \times \mathcal{T} \longrightarrow \mathbb{R},$$

with a vector of metric tools producing

$$\mathbf{m}(w,\tau) = \bigl(f_1(w,\tau), f_2(w,\tau), \ldots, f_n(w,\tau)\bigr) \in \mathbb{R}^n.$$

This formalization is central to the benchmark’s extensibility: the benchmark stores evidence, and metrics are versioned tools applied to that evidence (Ding et al., 7 May 2026).

3. Covered workloads, platforms, and configuration dimensions

Version 1.0 covers small-to-medium open-source models on GPU and TPU for training and batch inference. By the time of writing, the project had collected over 100 workloads spanning 7+ model architectures, 7 frameworks, and 3 hardware environments. The selected workload suite includes the following canonical examples (Ding et al., 7 May 2026):

ID	Workload
WL1	Qwen3-4B, inference, batch 128, input length 1024
WL2	Llama-3.1-8B, inference, batch 128, input length 1024
WL3	DeepSeek-MoE-16B, inference, batch 128, input length 1024
WL4	Llama-3.1-8B, training, batch 4, sequence 512
WL5	DeepSeek-V3-16B, training, batch 8, sequence 1024
WL6	DeepSeek-V3-16B, training, batch 64, sequence 2048
WL7	DeepSeek-V3-236B, training, batch 64, sequence 1024

The case studies primarily use NERSC Perlmutter and Google TPU v6e. Perlmutter is described as 4× A100 GPUs per node with NVLink 3.0 at 300 GB/s unidirectional bandwidth within node and Slingshot-11 at 200 Gbps between nodes. TPU v6e is described as 8 chips per node, 32 nodes per pod, a 2D torus, and 100 GB/s unidirectional ICI bandwidth. The software stack studied includes communication libraries such as NCCL, MSCCL++, and XLA collectives, and frameworks such as Megatron-LM, TorchTitan, MaxText, vLLM, and SGLang (Ding et al., 7 May 2026).

A distinctive feature of CCL-Bench is that it benchmarks the full configuration space rather than a single system axis. Hardware dimensions include accelerator type, memory hierarchy, scale-up interconnect bandwidth, scale-out network bandwidth, topology, and scale-up domain size. Software dimensions include framework, compiler stack, communication library, communication protocol, and collective-library tuning variables. Parallelism dimensions include DP, TP, PP, EP, CP, micro-batch size, pipeline schedule, activation recomputation, ZeRO stage, offload, and CUDA graphs. This suggests that the benchmark is designed not only to rank configurations but also to attribute performance to specific interactions among hardware, runtime, and parallelization choices (Ding et al., 7 May 2026).

4. Metric system and analytical methodology

The benchmark toolkit computes metrics across model execution, compute, memory, communication, and utility. Model-execution metrics include avg_step_time, mfu, ttft, and tpot. Compute metrics include mean_sm_coverage, dominant_kernel_concentration, compute_bound_fraction, memory_bound_fraction, and moe_fraction. Memory metrics include average_memory_bandwidth and memory_transfer_overhead. Communication metrics include communication_fraction, total_communication_time, compute_comm_overlap, load_imbalance_ratio, straggler, and effective bandwidth metrics for AllGather, AllReduce, ReduceScatter, AllToAll, and peer-to-peer traffic. Utility metrics include scale_up_bw_utility and related what-if metrics for scale-out bandwidth, scale-up bandwidth, scale-up domain size, and TPU ICI bandwidth (Ding et al., 7 May 2026).

Several of these metrics are explicitly defined. For a hardware resource $r$, utility is

$$\mathrm{Utility}(r) = \frac{T - T_{2\times}(r)}{T} \times 100\%,$$

where $T$ is the measured baseline step time and $T_{2\times}(r)$ is the simulated step time after doubling that resource. Compute-communication overlap is defined as

$$\text{Overlap} = \frac{\mathrm{union}(\mathcal{K}_\text{comm}) \cap \mathrm{union}(\mathcal{K}_\text{comp})}{\mathrm{union}(\mathcal{K}_\text{comm})} \times 100,$$

with $Y$0 denoting communication-kernel intervals and $Y$1 compute-kernel intervals in a step. Exposed memory-transfer overhead is defined as time spent in memory copies not overlapped with compute. These definitions matter because they turn traces into reusable quantitative objects rather than leaving them as purely diagnostic timelines (Ding et al., 7 May 2026).

The benchmark’s methodology also enables simulation-backed what-if analysis. Traces can be converted into Chakra execution graphs, in which compute kernels become COMP_NODEs, collectives become COMM_COLL_NODEs, idle intervals become gap nodes, and edges preserve stream order and rank-local sequencing. These graphs can then be replayed in Astra-Sim under modified network assumptions, allowing the benchmark to estimate end-to-end effects of doubling scale-up bandwidth, scale-out bandwidth, or TPU interconnect bandwidth without rerunning the original workload. A plausible implication is that CCL-Bench bridges direct measurement and counterfactual systems analysis more tightly than conventional leaderboard-oriented benchmarks (Ding et al., 7 May 2026).

5. Empirical findings and case studies

The benchmark’s main empirical contribution is that trace-backed evidence supports claims that summary-statistic benchmarks cannot. The first headline finding is that higher compute-communication overlap can coincide with longer training step time. This appears in DeepSeek-V3-16B MoE training (WL5), where higher overlap was associated with worse step time. Post-hoc metric extension showed that higher-overlap configurations used a smaller expert-parallelism domain relative to total GPU count, such as EP = 4 on 8 GPUs, whereas lower-overlap runs used EP = 8. The smaller EP domain increased expert replication across the data-parallel domain and therefore increased ReduceScatter and AllGather traffic for FSDP-style weight gathering and gradient synchronization. In other words, the extra communication volume outweighed the benefit of hiding more of that communication behind compute (Ding et al., 7 May 2026).

The second headline finding is that doubling TPU interconnect bandwidth yields far higher end-to-end utility than doubling GPU scale-up bandwidth on small and medium workloads. On Perlmutter A100, doubling scale-out bandwidth helps multi-node communication-heavy training by as much as 28.7%, while doubling scale-up domain size gives the largest gains for WL4 and WL5, at 53.9% and 46.7%, respectively. For large GPU workloads such as WL6 at 128 GPUs and WL7 at 256 GPUs, the utilities fall to 3.9% and 3.4%, indicating that these workloads are relatively compute-bound. On TPU v6e, by contrast, doubling ICI bandwidth yields up to $Y$2 the matched GPU scale-up bandwidth utility for inference and up to $Y$3 for training. The benchmark attributes this to TPU’s lower baseline interconnect bandwidth and lower-connectivity topology relative to the GPU scale-up fabric (Ding et al., 7 May 2026).

The third headline finding is that the best-tuned configuration on one training framework can run 3.4× slower than the best-tuned configuration on a peer framework on identical hardware. In the CCL-Search case study for WL4 on 16 Perlmutter GPUs, TorchTitan’s best configuration achieved step time 1.50 s with TP=1, DP=4, PP=4, while Megatron-LM’s best configuration achieved 0.44 s with TP=4, DP=1, PP=4. Applying TorchTitan’s best configuration to Megatron-LM yielded 1.3 s, which remained about 3× slower than Megatron-LM’s own optimum. CCL-Search reduced step time by up to 8× on TorchTitan and 19× on Megatron-LM relative to poor initial settings, and each trial was recorded as a full CCL-Bench entry. This indicates that framework-specific tuning is not a secondary concern but a primary determinant of measured infrastructure performance (Ding et al., 7 May 2026).

Additional case studies reinforce the benchmark’s diagnostic role. Holding workload, hardware, framework, parallelism, and accelerator count fixed, replacing NCCL with MSCCL++ reduced communication fraction from 49.5% to 46.5% for WL1 and from 53.7% to 46.4% for WL3, although latency gains were workload-dependent and WL2 regressed. Under identical A100 setups, vLLM was faster than SGLang on dense inference workloads WL1 and WL2 because it achieved substantially higher MFU, whereas SGLang was faster on MoE inference WL3 because it reduced exposed communication through a more efficient AllToAll. A TPU tensor-parallel sweep for Llama-3.1-8B inference showed a classic sweet spot: TP1 gave 222 ms, TP4 gave 85 ms, and TP8 regressed to 98 ms as communication fraction rose from effectively 0 to 19.3% and then 26.9% (Ding et al., 7 May 2026).

6. Limitations, extensibility, and naming distinctions

CCL-Bench 1.0 has explicit scope limitations. It covers only small-to-medium open-source models, only GPU and TPU, and only training and batch inference, and it does not include accuracy metrics. Trace storage is a practical constraint: a multi-rank trace can reach 10 GB, and very large runs can approach 100 GB. The benchmark also notes that traces may expose topology, operator sequences, or proprietary tuning choices, which can discourage sharing. Metric-only submissions are possible as a compromise, but they sacrifice auditability. The benchmark further states that no exhaustive coverage is possible because the workload–infrastructure cross product is too large by construction (Ding et al., 7 May 2026).

At the same time, extensibility is a central design objective. New metric tools can be added after traces have already been collected, validated by maintainers, and then run over all compatible historical entries. If a tool is later found incorrect, its version can be retracted and affected entries recomputed. Future versions may add larger models, more accelerator families such as Trainium, online serving with request-level latency, accuracy-affecting optimizations such as quantization, sparsity, and speculative decoding, more simulator plug-ins, performance-model integrations, and trace anonymization tooling. The benchmark is organized around a rolling, theme-driven submission model, which is intended to support growth without redesigning the schema (Ding et al., 7 May 2026).

The name also requires some disambiguation. CCL-Bench should not be conflated with CL-bench, which is a benchmark for context learning rather than LLM infrastructure (Dou et al., 3 Feb 2026). Nor should it be conflated with C$Y$4Bench, a benchmark for Classical Chinese understanding rather than systems evaluation (Cao et al., 2024). Within the benchmark literature, this naming overlap underscores a broader point: CCL-Bench’s distinctive contribution is not a new model task or linguistic evaluation set, but an evidence-centric benchmarking framework for infrastructure analysis, intended to shift LLM systems evaluation from ranking by outcomes to reasoning from traces (Ding et al., 7 May 2026).