---
title: 'SProBench: HPC Data Stream Benchmark'
url: https://www.emergentmind.com/topics/sprobench
type: topic
---

# SProBench: HPC Data Stream Benchmark

Searching arXiv for SProBench and related benchmark context.
SProBench is a benchmark suite for evaluating data stream processing frameworks on large-scale high performance computing (HPC) infrastructure. It is designed around the observation that, although frameworks such as Apache Flink, Spark Streaming, and Kafka Streams have advanced memory management, state handling, and fault-tolerance, scalability remains a first-class concern when ingest rates rise into the multi-gigabyte-per-second range. SProBench addresses the limited state of research on modern HPC systems by combining a modular architecture, native support for SLURM-based clusters, high-velocity workload generation, automated experiment management, and end-to-end metrics capture under realistic large-scale conditions [2504.02364].

## 1. Motivation and problem setting

SProBench is motivated by a gap between the scale of contemporary data stream processing workloads and the capabilities of existing measurement suites. The benchmark paper states that many benchmarks plateau at 1–4 M events/s and therefore fail to stress HPC-class clusters, even though modern systems comprise hundreds of nodes, thousands of cores, and low-latency interconnects [2504.02364]. In this setting, throughput and latency are constrained not only by framework internals but also by scheduler integration, broker behavior, and orchestration overhead.

A central premise is that local-machine and cloud-VM benchmarks do not adequately characterize HPC deployments. Existing suites are described as lacking native support for batch schedulers such as SLURM, using outdated framework bindings, and being unable to automatically orchestrate large-scale experiments. SProBench therefore treats SLURM integration, customizable processing pipelines, and high-velocity workload generation as benchmark requirements rather than deployment afterthoughts.

This scope distinguishes SProBench from narrower microbenchmarks or single-framework testbeds. It is not merely a data generator, nor only a workload suite for one execution engine. Instead, it is structured to measure generator, broker, processing framework, and system-level telemetry together, which is essential when the target environment is an HPC cluster rather than a standalone server.

## 2. Modular architecture

SProBench adopts a four-layer modular architecture in which components can be deployed independently, reconfigured via a single master YAML file, and chained together under SLURM [2504.02364]. The benchmark’s high-level organization separates the data path, telemetry path, and orchestration path.

| Layer/component | Elements | Function |
|---|---|---|
| Workload path | Workload Generator → Kafka Ingest → Processing Pipeline → Kafka Egress | End-to-end data flow |
| Metrics path | Metrics Collector (JMX + Pika/MetricQ) → Central Metrics Store | Runtime telemetry |
| Control path | CLI/SLURM Interface | Experiment orchestration |

The workload generator is a multi-threaded Java process that produces synthetic JSON events with a default structure of `{timestamp, sensorID, value}`. It supports constant-rate, random-interval, and burst-mode patterns, auto-scales the number of generator instances to meet a target throughput, reaches up to 0.5 M events/s per instance and more than 20 M events/s per node, and permits configurable event sizes of at least 27 bytes. The message broker is typically Apache Kafka, although the broker layer is pluggable; its architectural role is to decouple generation from processing while exposing configurable partitioning.

The stream processing layer provides three predefined pipelines. The Pass-Through pipeline performs no operations and serves as a baseline. The CPU-Intensive pipeline applies tuple parsing, unit conversion, and threshold checks. The Memory-Intensive pipeline uses sliding-window aggregate state over keyed streams. The metrics subsystem collects JVM-level counters such as CPU%, heap usage, and GC count/time via JMX, as well as system-level metrics including network I/O, memory bandwidth, disk I/O, and energy via Pika and MetricQ. Latency and throughput probes are placed at each pipeline boundary, and post-processing scripts aggregate, validate, and visualize the results.

## 3. Framework integration and execution model

SProBench provides native implementations for Apache Flink, Spark Streaming, and Kafka Streams, which the benchmark describes as “first-class citizens” [2504.02364]. This framework coverage is important because it enables comparison across established stream processing paradigms without changing the surrounding benchmarking infrastructure.

Integration occurs through deployment scripts and shared Java bindings. The deployment layer includes `flink-deploy.sh`, `spark-deploy.sh`, and `kafka-streams.sh`, each of which invokes the framework CLI with the job JAR, class name, parallelism, and checkpoint settings. The shared Java library defines input and output topics, serializers, and watermark strategies. This creates a consistent boundary between benchmark control and framework-specific job logic.

The execution model is explicitly SLURM-native. A single CLI entrypoint reads the YAML configuration, including nodes, cores, memory, parallelism, and workloads. In interactive mode it verifies local resources; in batch mode it emits an `sbatch` script. Environment variables, dependencies, job arrays, and logs are auto-managed. Under SLURM, each worker’s task receives CPUs and threads according to the configuration, while memory and network limits are passed through `--cpus-per-task` and `--mem`. The paper highlights parameter-sweep experiments such as `parallelism=1,2,4,8,16`, which indicates that reproducibility and batch exploration are core features rather than auxiliary scripts.

A plausible implication is that SProBench is intended for operational benchmarking campaigns in shared HPC environments, not only for isolated proof-of-concept measurements. The emphasis on job arrays, structured logs, and scheduler-managed resources supports that interpretation.

## 4. Metrics, formalization, and experimental methodology

SProBench formalizes four key quantities: throughput $T$ in events/s, latency $L$ in seconds, speedup $S(n)=\frac{T(n)}{T(1)}$, and overhead $O$ [2504.02364]. The benchmark assigns them distinct interpretive roles: throughput quantifies raw processing capacity, latency captures end-to-end delay, speedup reveals parallel scaling efficiency, and overhead isolates additional costs introduced by checkpoints, serialization, or framework overhead. Even without reducing evaluation to a single scalar score, these definitions provide a compact framework for analyzing scale-up behavior and trade-offs between capacity and delay.

The evaluation reported for SProBench was conducted on the Barnard cluster at TU Dresden. The cluster configuration is given as 630 nodes, each with 2× Intel Xeon Platinum 8470 processors for a total of 104 cores per node, 512 GB DDR5-4800 MT/s RAM, and an InfiniBand interconnect; SLURM v20 manages both batch and interactive jobs. Workloads use default sensor-data streams of 27 B per event, with rates varied from 0.5 M to 40 M events/s. Parallelism levels are 1, 2, 4, 8, and 16 threads or slots, and the benchmark exercises the pass-through, CPU-intensive, and memory-intensive pipelines.

Automation is organized around the master YAML configuration. The CLI automatically submits job arrays for each combination of pipeline, parallelism, and data rate, while capturing logs and metrics in structured directories. This methodology matters because it standardizes experiment structure across frameworks and scales, reducing the risk that scheduler artifacts or ad hoc deployment choices dominate the results.

## 5. Reported performance characteristics

The benchmark reports linear 1:1 scaling for the generator plus Kafka subsystem up to 40 M events/s across 32 generator instances and 4 Kafka partitions, with broker latency growing linearly while remaining within sub-10 ms per event [2504.02364]. This establishes that the ingress side is capable of stressing the downstream processing engines at rates beyond those commonly reported by earlier stream-processing benchmarks.

For the full Apache Flink pipeline on a single node, the CPU-Intensive pipeline reaches approximately $T \approx 2$ M evt/s with $L \approx 20$ ms at parallelism 1, and approximately $T \approx 14$ M evt/s with $L \approx 50$ ms at parallelism 16. The corresponding speedup is reported as $S(16)\approx 7.0$, characterized as sublinear because of GC and network contention. The Memory-Intensive pipeline shows similar scaling with slightly higher latencies. These results indicate that higher parallelism substantially increases throughput, but not without measurable latency growth and efficiency loss.

The paper also frames SProBench relative to prior benchmarks. Compared with the Yahoo Streaming Benchmark at 0.2 M evt/s and DSPBench at 0.8 M evt/s, SProBench’s peak single-node rate of approximately 20 M evt/s is more than 25× higher, and its multi-node aggregate of 40 M evt/s is more than 50× higher. The reported bottlenecks include generator CPU saturation and Kafka network I/O, which were addressed through multithreading, optimized I/O threads, and SLURM-driven co-scheduling. The benchmark therefore presents scalability not as a property of the processing framework alone, but as an end-to-end systems property involving workload generation, message transport, resource placement, and runtime coordination.

A common misunderstanding would be to interpret these numbers as pure framework scores. The benchmark’s own structure argues against that interpretation: the measured outcome depends on the coordinated behavior of generator, broker, framework, and cluster scheduler.

## 6. Extensibility, licensing, and position in the benchmark landscape

SProBench is explicitly designed for customization. New workloads can be introduced by defining a new JSON schema, event size, and arrival pattern through a custom Java plugin. New pipelines can be added by dropping in Flink or Spark job JARs or Kafka Streams applications via documented interfaces. The metrics subsystem can be extended with external probes such as Lustre I/O and GPU utilization by extending the Prometheus-style collector. Resource configuration is likewise open-ended: any SLURM parameter, including QoS, `gres`, and GPUs, can be set in the master configuration [2504.02364].

The software is released as open source at `https://github.com/apurvkulkarni7/SProBench` under the Apache 2.0 license, with contributions handled through a fork-and-PR model. In practical terms, this licensing and deployment posture makes the suite suitable for reuse in academic benchmarking campaigns and infrastructure-specific extensions.

In the broader benchmark landscape, SProBench belongs to the class of domain-specific benchmark suites that combine synthetic data generation with carefully scoped measurement objectives. SP²Bench, for example, was introduced as a publicly available SPARQL performance benchmark with a deterministic, platform-independent data generator and design goals centered on relevance, portability, scalability, and understandability [0806.4627]. This suggests that SProBench occupies a comparable methodological niche for HPC-oriented stream processing: it is not a universal benchmark for all data systems, but a benchmark suite whose value derives from the alignment between workload model, execution environment, and evaluation criteria.

Its principal significance lies in filling the HPC-scale stream processing evaluation gap identified by its authors. The benchmark’s combination of SLURM-native orchestration, framework interoperability, customizable pipelines, and end-to-end telemetry positions it as an infrastructure-aware measurement framework rather than a narrow application benchmark [2504.02364].

Source: https://www.emergentmind.com/topics/sprobench