CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructure

Abstract: Evaluative claims about LLM infrastructure -- ``workload X is fastest on hardware Y with software Z'' -- depend on a complex configuration space spanning hardware accelerators, interconnect bandwidth, software frameworks, parallelism plans, and communication libraries. Current infrastructure evaluation benchmarks publish a small set of end-to-end numbers that do not explain why one configuration outperforms another. We present CCL-Bench, a trace-based benchmark that addresses the limitations of existing benchmarks by recording reusable evidence for every ML workload. Each contributed data point in CCL-Bench packages an execution trace, a YAML workload card, and the launch scripts. We have developed a community-extensible toolkit to compute fine-grained compute, memory, and communication efficiency metrics from this evidence. Using CCL-Bench, we surface three claims that summary-statistic benchmarks cannot support: (i) higher compute-communication overlap can coincide with longer training step time and reveal inefficient parallelization choices, (ii) doubling TPU interconnect bandwidth yields a much higher end-to-end improvement in step time than doubling GPU interconnect bandwidth on small and medium workloads, and (iii) the best-tuned configuration on one training framework can run up to 3$\times$ slower than the best-tuned configuration on a peer framework on identical hardware.

• The paper presents a trace-centric framework that standardizes execution traces, workload cards, and launch scripts to deliver detailed, reproducible LLM performance insights.
• The methodology reveals that increased compute-communication overlap can inadvertently raise latency when optimal configurations and framework-specific tuning are not employed.
• By leveraging trace-derived metrics and simulation-grounded analyses, CCL-Bench enables evidence-based evaluation of hardware upgrades and configuration optimizations.

CCL-Bench 1.0: Trace-Based Benchmarking for LLM Infrastructure

Introduction and Motivation

Evaluating LLM system performance across the rapidly evolving hardware and software landscape presents significant attribution and reproducibility difficulties stemming from enormous configuration and metric spaces. Legacy infrastructure benchmarks such as MLPerf and LLM-Perf provide only coarse, metric-frozen, end-to-end summaries that do not support explanatory diagnosis or verifiable comparison beyond outcome-level ranking. They fail to expose root causes behind observed performance gaps, limit the post-hoc introduction of new metrics, and lack visibility into configuration tuning effort.

CCL-Bench 1.0 (2605.06544) directly targets these limitations via a trace-centric benchmarking framework that offers fine-grained, cross-stack evidence for each workload run. By standardizing trace, workload, and launch provenance artifacts—and providing a community-extensible metric analysis toolkit—it enables outcome explanations, retroactive metric extensions, simulation-grounded what-if analyses, and rigorous, reproducible configuration optimization.

Figure 1: CCL-Bench overview: a standardized trace plus workload card recording each run, a metric toolkit computing fine-grained metrics, and downstream analysis and optimization plug-ins.

CCL-Bench Methodology

Artifact Schema and Collection

Each CCL-Bench submission consists of three required, auditable components:

• Execution Trace: A raw kernel-level trace (Kineto/NSYS for GPU or XProf for TPU) capturing all operator, communication, and memory activities with per-rank granularity, covering kernels, collectives, timestamps, and relevant hardware counters.
• Workload Card: A YAML record specifying all experimental context, including model, batch/sequence parameters, model family and size, hardware architecture (XPU type and network), system stack (training/serving framework, compiler, comm library, parallelism config), and provenance.
• Launch Scripts: The scripts used for trace generation, ensuring run reproducibility.

The schema supports rapid community growth and the precise mapping of any post-hoc metric or simulation to the corresponding infrastructure axes.

Metric Toolkit

Metrics in CCL-Bench are decoupled from experiment time, with trace-resident evidence making new analyses universally retroactive. The initial toolkit covers step time, Model FLOPs Utilization (MFU), compute-comm overlap, comm and DMA breakdown, MoE fraction, peak bandwidths, and utility of hardware upgrades, among others.

Figure 2: (a) MFU vs. step time and compute-comm overlap vs. step time on A100 GPU (Perlmutter). Each point is one CCL-Bench entry. (b) Step-time and communication traffic-volume breakdown for WL5 DeepSeek-V3-16B EP=4 and EP=8 MoE training runs.

The tools permit slicing, configuration attribution, and causal diagnosis that are infeasible with pre-frozen, summary-only leaderboards. Tool logic and versioning are explicitly recorded; artifacts can be challenged or revised via public, code-reviewed interface.

Interaction Model and Submission Workflow

Contributors (vendors, framework authors, operators) submit evidence for a standardized pool, validated by rule- and schema-based checks. The results table allows selective workload/hardware slicing for cross-infrastructure studies. To access raw traces (which may encode sensitive topology/configuration data), users must reciprocate with a compatible submission, incentivizing pool growth while protecting proprietary interests.

Figure 3: CCL-Bench user interface. The interface provides a dashboard for performance ranking, workload comparison, and trace upload.

Major Findings and Technical Claims

Compute-Communication Overlap is Non-Indicative of Step Time

CCL-Bench evidence reveals that simply maximizing compute-comm overlap does not guarantee improved step latency. For example, DeepSeek-V3-16B training shows higher overlap can coincide with greater communication volume and increased latency when parallelism domain choices are suboptimal. A higher expert parallelism degree (EP=8) reduces redundant collective traffic compared to a smaller EP (EP=4), counteracting naive overlap metrics:

Figure 2: (repeated) MFU/step time and comm traffic-volume breakdown demonstrating inefficiency of maximizing overlap without regard to global comm volume.

Claim: Higher computation-communication overlap can increase LLM training latency by driving collective inefficiencies if parallelism is poorly chosen.

Hardware Upgrade Utility Analysis

By exporting trace-derived kernel/collective graphs into workload simulators (Astro-Sim/Chakra), CCL-Bench enables quantification of the incremental utility of hardware upgrades per real workload. The study shows that doubling TPU inter-chip interconnect (ICI) bandwidth confers substantially larger step-time reductions than equivalent GPU NVLink upgrades for both training and inference at corresponding scale and batch size:

Figure 4: Utility metrics for scale-out bandwidth, scale-up/ICI bandwidth, and scale-up domain size on Perlmutter A100 Slingshot and TPUv6e Torus.

For small and medium workloads, doubling TPU ICI bandwidth yields up to 100× larger reduction in step latency compared to GPU NVLink upgrades (inference) and up to 22× improvement for training.

Claim: The performance impact of network upgrades is platform- and workload-dependent: incremental bandwidth on TPUs is substantially more valuable than on GPUs for certain regimes, while GPU utility saturates until scale-up domain is also increased.

Framework-Specific Configuration Optima

CCL-Bench's LLM-agent-driven configuration optimizer, CCL-Search, records all steps of the search process alongside final outcomes. When benchmarking TorchTitan and Megatron-LM on the same hardware and workload, best-found configurations diverge drastically, and cross-applying the optimum configuration from one framework to the other can leave up to a 3× performance gap:

Figure 5: CCL-Search results. Bottom panels show explored TP, DP, PP, micro-batch size, and activation-checkpointing choices. The agent finds up to 8× and 19× speedups within 15 iterations and illuminates framework-specific configuration optima.

Claim: Optimal configuration is non-transferable for distributed LLM frameworks; reported leaderboard wins not paired with verifiable, trace-recorded tuning trajectories are not meaningful for infrastructure comparison.

Extended Analyses

Cross-System and Cross-Architecture Insights

Pairwise matched experiments dissect the sources of framework and communication stack performance differential. For identical hardware and workload, vLLM achieves higher MFU and hence faster dense model inference than SGLang, while SGLang’s optimized AllToAll delivers lower communication fraction and faster MoE inference for DeepSeek-MoE-16B. On training workloads, TPU consistently delivers higher MFU and lower step time than A100 due to both architectural differences and reduced communication bottlenecking, but inference and small-batch results are more nuanced.

Figure 6: Example cross-system comparisons with all axes controlled except the communication stack.

Figure 7: Example cross-system comparisons with all axes controlled except the framework, demonstrating the attributional diagnostic capability of CCL-Bench.

Trace-Driven Simulation: Networking Diminishing Returns

Sweeps increasing network resources for 236B parameter, 256-GPU workloads reveal diminishing returns of communication overlap as bottlenecks shift between software-layer scheduling and physical transport, quantitatively informing the utility of low-latency fabric improvements.

Figure 8: Cluster architecture sweep for DeepSeek-V3-236B, showing effect of combined scale-up/domain/bandwidth upgrades on overlapped communication.

Theoretical and Practical Implications

CCL-Bench establishes an evidence-based paradigm for LLM infrastructure benchmarking, enabling:

• Retrospective Metric Introduction: Any new efficiency, utilization, or load imbalance metric can be applied to legacy runs, immediately broadening diagnostic coverage.
• What-if Simulation Grounding: Traces enable simulation-based analyses with fidelity impossible for analytics-only models, facilitating principled hardware and topology co-design.
• Reproducible, Audited Tuning: Tuning trajectories are fully archived, supporting fair, apples-to-apples cross-framework and cross-vendor benchmarks.
• Framework-Agnostic Leaderboards: By decoupling metric evaluation from the system under test, the platform moves the field from authority-led scoreboards to community-audited, analysis-rich grounds.

Practically, this transforms infrastructure evaluation from opaque, one-dimensional record-keeping to a multidimensional, collaborative, and theory-driven discipline.

Limitations and Future Developments

Current coverage is limited to open-weight models, training and batch inference workloads, and excludes some accelerators (Trainium, HPU) and online serving. Trace storage introduces privacy/commercial sensitivity challenges, necessitating policies for anonymization or entropy-conserving compression. The configuration cross-product remains impossibly large; submission diversity and rolling workload axes aim to maximize diagnostic value.

Future directions include integrated plug-ins for additional simulators, automated trace-space summarization for leaderboard scale, and increased vendor participation to extend configurational and architectural breadth.

Conclusion

CCL-Bench 1.0 moves LLM infrastructure benchmarking towards a reproducible, evidence-grounded, and extensible regime. By decoupling metric analyses from experimental time and enabling post-hoc, community-verified explanation, the framework facilitates rigorous infrastructure comparison and bottleneck attribution, guides system and hardware optimization, and enables both operators and researchers to evaluate claims on a shared foundation. The methodologies and tools demonstrated here are positioned to become foundational infrastructure for the next generation of AI system benchmarking, where explanation and reproducibility are as critical as performance outcomes.