SAKURAONE: Open HPC Cluster

• SAKURAONE is a managed HPC cluster that supports advanced AI workloads with a fully open, vendor-neutral 800 GbE Ethernet fabric.
• The system’s architecture integrates 800 GPUs, 150 TB DRAM, and multi-petabyte NVMe storage to achieve top-tier benchmark performance including 33.95 PFLOP/s in HPL.
• Empirical studies reveal a workload mix from small, I/O-bound tasks to massive LLM trainings, underscoring the need for flexible scheduling and real-time observability.

SAKURAONE is a managed high performance computing (HPC) cluster developed and operated by the SAKURA Internet Research Center and optimized for advanced workloads, including LLM training. It builds on the KOKARYOKU PHY bare metal GPU platform and, in ISC 2025 TOP500, was ranked 49th by HPL with measured performance of 33.95 PFLOP/s (HPL Rmax), 396.295 TFLOP/s (HPCG), and 339.86 PFLOP/s on HPL-MxP with FP8. Across the 2025 and 2026 descriptions of the system, SAKURAONE is distinguished by a fully open networking stack based on 800 GbE, SONiC, and RoCEv2, and by an operational study of workload dynamics under exclusive use by a single research project (Konishi et al., 15 Apr 2026, Konishi, 2 Jul 2025).

1. Institutional setting and system role

SAKURAONE is presented as a private-sector HPC resource in Japan, operated as a managed cluster by the SAKURA Internet Research Center. The 2025 account situates it in a national context in which academic and public HPC resources in Japan, including ABCI3.0 and TSUBAME4.0, are shared with academia, making stable, commercial-scale access difficult. It further reports a two-stage investment of approximately ¥13 billion in 2023 plus a ¥100 billion government-supported project in 2024, framing SAKURAONE as a component of Japan’s next-generation AI and cloud infrastructure (Konishi, 2 Jul 2025).

The system is also characterized as an open and transparent AI platform. In that framing, SAKURAONE is not only a compute resource but also a demonstration that vendor-neutral Ethernet-based networking can be deployed at a scale recognized by the TOP500. A plausible implication is that the project serves simultaneously as production infrastructure and as an empirical case study for open HPC system design.

2. Compute and storage architecture

SAKURAONE consists of 100 identical GPU-accelerated compute nodes arranged in two identical pods. Each node contains 2 × Intel Xeon Platinum 8580+ processors, 1.5 TB DDR5-5600 memory, and 8 × NVIDIA H100 SXM 80 GB GPUs connected by an NVLink/NVSwitch fully connected GPU mesh. Local storage is provisioned as 4 × 7.68 TB NVMe SSD for scratch plus 2 × 372 GB SAS disks for the system disk, and the aggregate hardware totals reported for the cluster are 800 GPUs, 150 TB DRAM, and approximately 3.07 PB of local NVMe scratch (Konishi et al., 15 Apr 2026, Konishi, 2 Jul 2025).

The shared storage subsystem is an all-flash Lustre file system. The 2026 description reports four DDN ES400NVX2-NDR200 front-end servers, each with dual Ice-Lake CPUs and 24 × 30.72 TB TLC SSD bays, while the 2025 description characterizes the same backend class as 4 × DDN ES400NVX2-NDR200 with dual active controllers, Ice Lake CPUs, and 24 PCIe Gen4 NVMe drives. Capacity is reported as 2 PB, with the 2026 paper specifying 2 PB usable as 1 PB data plus 1 PB safety margin, and the 2025 paper specifying 2 PB physical. Throughput is described as approximately 100 GB/s sustained for simultaneous multi-node checkpoint writes and data generation, and as 200 GB/s theoretical peak read and write in the storage benchmark-oriented description (Konishi et al., 15 Apr 2026, Konishi, 2 Jul 2025).

The system-level organization reflects the KOKARYOKU PHY bare-metal GPU server model. In the software and management description, that term refers to the two-socket, eight-GPU, NVLink/NVSwitch super-server design, with fine-grained PCIe/NIC affinity derived from nvidia-smi topo analysis to ensure NUMA-local traffic for MPI and NCCL over RoCEv2 (Konishi, 2 Jul 2025).

3. Open Ethernet fabric and rail-optimized topology

The network is a rail-optimized 800 GbE leaf-spine fabric. SAKURAONE is divided into two pods of 50 nodes each; each pod contains eight leaf switches, and both pods connect to a common spine layer of eight switches via 800 GbE inter-switch links. Each node presents 8 × 400 GbE NIC rails for the GPU fabric, with each of those links fanning out to the pod’s eight leaf switches, yielding full bisection bandwidth and uniform shortest-path connectivity across pods (Konishi et al., 15 Apr 2026, Konishi, 2 Jul 2025).

The switch platform is based on Edgecore white-box hardware with Broadcom Tomahawk 5 ASICs, and the network operating system is SONiC under a disaggregated SAI control plane. The transport is RoCEv2 with Priority Flow Control and ECN, including DCQCN tuning. The stated rationale is that a fully open Ethernet stack avoids vendor lock-in, leverages the community ecosystem for rapid feature evolution, and has been shown to match proprietary interconnect performance when properly tuned (Konishi et al., 15 Apr 2026).

The rail-optimized design is motivated by collective communication behavior. Because nodes expose multiple independent NIC rails, collective libraries such as NCCL can stripe across rails to reduce congestion during synchronized bursts. The 2025 description adds that compute-network and storage-network traffic are fully separated for stability, with each node using 2 × 400 GbE links for Lustre I/O. A recurring point of debate in HPC concerns whether Ethernet-based fabrics can substitute for proprietary interconnects such as InfiniBand or Cray Slingshot; SAKURAONE is presented as evidence that a fully open, vendor-neutral stack can sustain top-tier benchmark performance while preserving operational flexibility (Konishi, 2 Jul 2025, Konishi et al., 15 Apr 2026).

4. Benchmark performance and efficiency model

The reported benchmark results are summarized below.

Benchmark	Configuration	Reported result
HPL	784 GPUs, $N=2{,}706{,}432$, $P \times Q = 16 \times 49$	$R_{\max} = 33.95$ PFLOP/s
HPCG	784 processes, $\sim 55.9$ billion unknowns, 1.51 trillion nonzeros	396.295 TFLOP/s
HPL-MxP	768 GPUs, sloppy FP8, $N=2{,}989{,}056$, $24 \times 32$ grid	$R_{\max} = 339.86$ PFLOP/s

For HPL, the sustained result is 33.95 PFLOP/s on 784 GPUs, corresponding to 43.31 TFLOP/s per GPU in 389.23 s. The system’s rank is 49th worldwide in the ISC 2025 TOP500 list, and it is described as the only top-100 system with a fully vendor-neutral, open Ethernet-based interconnect stack built from SONiC and RoCEv2. The 2026 paper gives the efficiency model as

$$P_{\mathrm{peak}} = N_{\mathrm{GPUs}} \times F_{\mathrm{clock}} \times \mathrm{FLOP \cdot per \cdot cycle},$$

and notes that each H100 GPU with 132 SMs at 1.98 GHz can deliver up to approximately 55.34 TFLOP/s in DP GEMM. On that basis, HPL per-GPU efficiency is reported as approximately $43.31 / 55.34 \simeq 78.3\%$, and overall HPL cluster efficiency as approximately $$33.95 \text{ PFLOP/s} / (784 \times 55.34 \text{ TFLOP/s}) \simeq 78\%$$ (Konishi et al., 15 Apr 2026).

The 2025 paper also provides the standard HPL operation-count model, with $P \times Q = 16 \times 49$0, and writes the sustained rate as

$P \times Q = 16 \times 49$1

For HPCG version 3.1, the reported problem grid is $P \times Q = 16 \times 49$2, corresponding to 55.9 billion equations and 1.51 trillion nonzeros, with approximately 39.96 TB total memory used, of which 35.17 TB is for the linear system. Observed peak bandwidth is 3.316 TB/s, and the final result is 396.295 TFLOP/s, noted as approximately 0.8% of HPL peak; this is explicitly interpreted as a sparse-matrix, memory- and communication-bound measure (Konishi, 2 Jul 2025).

For HPL-MxP version 25.4.0, using sloppy FP8 (type 1), SAKURAONE achieved 339.86 PFLOP/s on 768 GPUs, or 442.5 TFLOP/s per GPU, with an LU-only phase peak of 539.19 PFLOP/s, or 702.07 TFLOP/s per GPU. The reported validation residual is approximately $P \times Q = 16 \times 49$3, below $P \times Q = 16 \times 49$4, and the 2025 description states that the FP8 result is approximately $P \times Q = 16 \times 49$5 faster than the FP64 HPL result when expressed at the exaflop scale as $P \times Q = 16 \times 49$6 versus $P \times Q = 16 \times 49$7 EFLOP (Konishi, 2 Jul 2025).

The 2025 paper additionally reports I/O500 results on Lustre, comparing 96-node and 10-node configurations. The bandwidth score is 139.80 GiB/s versus 133.03 GiB/s, metadata IOPS are 327.84 kIOPS versus 248.74 kIOPS, and the total IO500 score is 214.09 versus 181.91. This suggests that the storage system was engineered not only for checkpoint throughput but also for metadata-intensive production workloads (Konishi, 2 Jul 2025).

5. Software environment and orchestration

The base operating system is Rocky Linux 9.4 (Blue Onyx), described as RHEL-compatible and community-driven. The core software environment includes GCC, CUDA 12.x series, cuDNN 8.9–9.6, TensorRT, and NCCL 2.20+. The job scheduler is Slurm 22.05.9 with multi-project support, QoS, reservations, and accounting. The MPI stack is HPE HPC-X 2.17/2.18 in GCC+CUDA builds, including debug, profiled, and multithreaded variants (Konishi, 2 Jul 2025).

Container and ML tooling are also explicitly part of the platform definition. SAKURAONE supports Singularity-CE 4.3.1, identified parenthetically as Apptainer, and Pyxis for HPC containers, alongside Miniconda-based Python environments with PyTorch and TensorFlow builds. The 2025 description states that deep learning frameworks are pre-installed and that NCCL plus MPI over RoCEv2 are used for collectives such as all-reduce and broadcast in large-scale data-parallel training. Container-based reproducibility is described as supporting rapid spin-up of multi-node training jobs (Konishi, 2 Jul 2025).

This software profile is significant because it aligns the system with both traditional HPC and modern LLM development practice. Rather than treating the cluster as a benchmark-only platform, the papers describe an environment intended for iterative experimentation, large distributed training, storage-intensive checkpointing, and containerized reproducibility.

6. Observed workload dynamics in a single-tenant LLM environment

The 2026 paper studies SAKURAONE under exclusive use by a single research project and reports workload characteristics for development-related jobs from December 2024 through March 2025. The job-size distribution is strongly skewed: 76.9% of jobs used a single node and 86.4% used 1–4 nodes, yet those small jobs accounted for only 1.8% and 4.6% of total GPU-occupied time. By contrast, jobs using 17 or more nodes constituted 3.3% of job count but consumed 73.3% of GPU-occupied time. The paper explicitly identifies this as a long-tail pattern in which many small experiments coexist with a few large-scale trainings that dominate time (Konishi et al., 15 Apr 2026).

GPU utilization varies sharply by job scale. Large-scale jobs spanning 17–32 nodes had median GPU utilization of approximately 98.4% and spent approximately 1.1% of time below 20% utilization. Mid-scale jobs spanning 3–16 nodes showed median utilization from 42% to 92%, with moderate spread. Small-scale jobs spanning 1–2 nodes had median utilization of 17.7% to 23.4% and spent 69% to 76% of time below 20% utilization. The paper attributes the low utilization in small jobs to I/O- or CPU-bound tasks such as data preparation and evaluation, which are common in LLM workflows (Konishi et al., 15 Apr 2026).

The temporal evolution of resource use mirrors the development lifecycle of an LLM project. In the early phase, daily 17–32 node continued pretraining jobs dominated. From mid-February 2025 onward, the system saw a gradual rise in 3–16 node fine-tuning and evaluation jobs. The paper states that this shift mirrors bulk pretraining followed by iterative, lower-scale tuning and evaluation. This suggests that static assumptions about “typical” cluster utilization can be misleading when the tenant is a unified research program rather than a heterogeneous multi-user population (Konishi et al., 15 Apr 2026).

Operationally, the workload study also reports that user-initiated cancellations accounted for 73.5% of GPU-occupied time. The interpretation given is that practitioners set high maximum steps and terminated runs early upon convergence, thereby improving overall efficiency. Failed jobs were 16.9% of job count but only 0.3% of GPU time, indicating that most failures occurred early. On that basis, the paper recommends checkpoint-based preemption so that transient short jobs can run during long-training intervals and then allow the large job to resume; it also emphasizes elastic reconfiguration and real-time observability, including user-driven cancellations guided by loss curves, as mechanisms for maximizing utilization in single-tenant LLM projects (Konishi et al., 15 Apr 2026).

7. Significance for open HPC and AI infrastructure

SAKURAONE’s primary significance in the cited literature lies in the conjunction of three claims: Top500-recognized performance, a fully open Ethernet-based interconnect stack, and empirical workload observations from a real LLM development environment. The system is repeatedly described as the only top-100 system using a fully open SONiC-based 800 GbE Ethernet stack, and the benchmark results are used to argue that vendor-neutral technologies can match proprietary HPC networks when properly tuned (Konishi, 2 Jul 2025, Konishi et al., 15 Apr 2026).

The broader research implications presented in the 2025 paper are correspondingly architectural and organizational. Vendor neutrality is said to foster customization, reduce lock-in, and leverage community-driven network operating system development. The coupling of compute, storage, and network fabrics, together with logical traffic separation, is framed as a lesson for predictable performance. Mixed-precision benchmarking, especially HPL-MxP, is described as essential for future AI-focused systems, while containerization through Singularity and Pyxis and scheduling through Slurm are identified as critical for supporting diverse and evolving research workloads (Konishi, 2 Jul 2025).

At the same time, the workload analysis introduces a qualification to simplistic utilization narratives. The prevalence of many low-utilization small jobs does not, by itself, indicate poor cluster design, because those jobs may correspond to evaluation, data preparation, or other I/O- and CPU-bound phases that are intrinsic to LLM development. Conversely, the high efficiency of large-scale training jobs does not eliminate the need for flexible scheduling and observability. A plausible implication is that SAKURAONE is best understood not merely as a benchmarked machine, but as a case study in aligning open HPC architecture with the lifecycle structure of single-project LLM research (Konishi et al., 15 Apr 2026).