Semantic Shuffle Benchmark

• The paper introduces ShuffleBench, which isolates the shuffle+aggregate stage to measure the performance of distributed stream processing frameworks.
• It presents a formal workload model with configurable tuple schemas, adjustable workload parameters, and detailed metrics such as throughput and latency quantiles.
• The benchmark integrates with platforms like Flink, Hazelcast Jet, Kafka Streams, and Spark, highlighting clear trade-offs between scalability, throughput, and latency.

ShuffleBench is a benchmark for large-scale data shuffling operations in distributed stream processing frameworks (DSPFs). It targets a workload class in which DSPFs are used primarily for shuffling, that is, re-distributing, data records so that stateful “real-time consumers” can execute state-local aggregations, while the aggregation logic itself is treated as a black-box software component. The benchmark was introduced from requirements in near real-time analytics for a large cloud observability platform, but it is explicitly designed to be domain-independent through a synthetic load generator, adjustable tuple schema, and configurable workload parameters. It is distributed as ready-to-use open-source software with Kubernetes-based orchestration and implementations for Apache Flink, Hazelcast Jet, Kafka Streams, and Spark Structured Streaming (Henning et al., 2024).

1. Workload class and benchmarking rationale

ShuffleBench addresses a gap in existing stream-processing benchmarks. The paper contrasts it with benchmarks oriented toward traffic analytics or IIoT sensor processing and argues that they do not cover a general use case in which a very large number of independent stateful “real-time consumers” must each receive small subsets of a high-volume stream. In that setting, the dominant systems problem is large-scale data shuffling, or re-partitioning, coupled with state-local aggregation, rather than complex joins or window-functions (Henning et al., 2024).

The motivating use case is cloud observability analytics. A cloud observability platform allows clients to register thousands to millions of continuous queries, described as “real-time consumers,” that maintain per-query state such as anomaly detection or sliding-window metrics. Each incoming telemetry record must be routed to exactly those query-consumers whose predicates match, and state updates must be local to each consumer instance to avoid cross-instance coordination. This formulation makes the shuffle step central: records are duplicated according to predicate matches and then re-partitioned by consumer identity.

A common misconception in DSPF evaluation is that benchmarking should primarily center on query semantics such as joins, windows, or SQL layers. ShuffleBench instead isolates the shuffle+aggregate stage as the core workload. This reflects the paper’s claim that DSPFs are especially relevant here because they provide scalable cluster management, fault-tolerant keyed state, and consistency guarantees, even when the aggregation logic is externalized as a black box.

2. Formal workload model

ShuffleBench specifies each input record as a tuple $(r.id, r.ts_{in}, r.payload)$, where $r.id$ is a unique identifier, $r.ts_{in}$ is a Kafka-assigned timestamp at ingestion, and $r.payload$ is an opaque byte array of configurable length $S$, for example $128$–$1024\,\mathrm{B}$ (Henning et al., 2024).

Its processing topology has five stages. Records are first read from a Kafka topic with $P$ partitions. A matcher implemented as flatMap duplicates each record for each matching consumer $q \in Q$, producing intermediate tuples keyed by $q$. A shuffle step implemented as groupBy or keyBy then repartitions the stream on the consumer key so that all events for a given $r.id$0 are routed to the same instance. The aggregator is a stateful operator that maintains a state $r.id$1 for each consumer and may emit output events. Finally, the sink writes output events with appended timestamps $r.id$2 back into Kafka so that latency can be measured end to end.

The benchmark formalizes throughput and latency explicitly. Given $r.id$3 processed records over elapsed wall-clock time $r.id$4, throughput is defined as

$r.id$5

For each output event $r.id$6, latency is defined as

$r.id$7

Overall latency is summarized by quantiles, specifically $r.id$8, $r.id$9, and $r.ts_{in}$0.

Several workload parameters are exposed because they directly affect the behavior of the matcher, shuffle stage, state backend, and sink. The number of consumers $r.ts_{in}$1, for example $r.ts_{in}$2–$r.ts_{in}$3, determines state-table size and matcher work. The total selectivity

$r.ts_{in}$4

for example $r.ts_{in}$5, $r.ts_{in}$6, or $r.ts_{in}$7, controls shuffle volume. Record size $r.ts_{in}$8 affects network and serialization costs. State size per consumer can be increased to simulate larger state footprints. Output frequency $r.ts_{in}$9, such as emitting every $r.payload$0th update, influences sink volume and the observed end-to-end latency.

3. System architecture and framework integrations

The benchmark implementation consists of three principal components: a stateless Load Generator that emits records into Kafka at a target rate $r.payload$1, a DSPF Application deployed as a Kubernetes Deployment with replicas for a chosen framework, and a Latency Exporter that consumes the output topic and computes $r.payload$2 distributions offline (Henning et al., 2024).

Kubernetes orchestration is central to the benchmark’s reproducibility and fairness claims. Each DSPF runs in its own namespace and is managed by Theodolite scripts. Resource limits are set at Pod level, and the Kafka cluster is also containerized for in-cluster I/O as a 3-broker deployment. Theodolite is used not only for orchestration but also for scalability metrics, which ties the benchmark to a broader performance-engineering methodology rather than an ad hoc measurement script.

Framework-specific integrations implement the same logical topology through each system’s native API surface.

Framework	Integration
Flink	DataStream API flatMap → keyBy → AggregateFunction
Hazelcast Jet	Pipeline.flatMap + groupByKey + aggregate(UpsertP function)
Kafka Streams	KStream.flatMap → groupByKey → aggregate with in-built state stores
Spark	readStream → flatMapGroupsWithState → writeStream in micro-batch with configurable maxOffsetsPerTrigger

The core shuffle operator pseudocode given in the paper makes the benchmark’s semantics explicit. The source stream is flatMap-expanded by matcher.match(r) into per-consumer events, repartitioned via keyBy(e -> e.cid), and passed into an aggregation stage that invokes RealTimeConsumer.update(s_prev, e). Output emission is conditional on a counter modulus, s_out.count % k == 0, which operationalizes the configurable output frequency.

4. Metrics and evaluation protocol

ShuffleBench adopts metrics and methods from performance-engineering practice rather than restricting itself to raw throughput. Throughput is measured in two forms: ad-hoc throughput, where $r.payload$3 is fixed and committed offsets per second are measured, and sustainable throughput, where a binary search on $r.payload$4 is performed under a stability criterion. End-to-end latency is reported through $r.payload$5 quantiles of $r.payload$6. Scalability is evaluated with Theodolite’s Resource Demand, defined as cores per additional $r.payload$7, and Load Capacity, defined as maximum $r.payload$8 per core count. Resource utilization is also collected as CPU and memory per Pod through the Kubernetes API (Henning et al., 2024).

The baseline experimental protocol is highly specific. The cluster is AWS EKS with 10 nodes: 3×m6i.xlarge for DSPF, 3×m6i.2xlarge for Kafka, and 4×m6i.xlarge for load and benchmark infrastructure. The baseline parallelism is $r.payload$9, described as 9×1 vCPU, 4 GB each, and the generator rate $S$0 is varied from $S$1 to $S$2S = 1\,024\,\mathrm{B}$S$3|Q| = 10^6SS4\sigma_{total} = 20\%$$, and output emission every 10 updates. Each trial runs for 15 minutes in steady state and is repeated three times for confidence. Data collection combines Kafka offsets and timestamps with Pod CPU and memory exported through Prometheus.</p> <p>The distinction between ad-hoc and sustainable throughput is especially important. The reported experiments show that short-run peak rates can overstate stable operating points, which is a standard systems concern but here becomes part of the benchmark definition itself. This suggests that ShuffleBench is intended not merely as a comparative microbenchmark but as a framework for operationally meaningful capacity estimation.</p> <h2 class='paper-heading' id='baseline-performance-and-trade-offs'>5. Baseline performance and trade-offs</h2> <p>Under the baseline configuration, ShuffleBench reports the following median throughput values over three runs (<a href="/papers/2403.04570" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Henning et al., 2024</a>).</p> <div class='overflow-x-auto max-w-full my-4'><table class='table border-collapse w-full' style='table-layout: fixed'><thead><tr> <th>Framework</th> <th style="text-align: right">Ad-hoc$$S$5 (r/s) Sustainable $S$6 (r/s) Flink 770,000 650,000 Kafka Streams 550,000 510,000 Hazelcast Jet 260,000 260,000 Spark (1 M max) – 280,000

The paper states that Flink leads by approximately $S$7 over Kafka Streams, while Hazelcast Jet reaches approximately $S$8 of Flink’s rate. It also reports that ad-hoc throughput overestimates sustainable throughput by $S$9–$128$0 for Flink and Kafka Streams, whereas Hazelcast is unaffected in this baseline.

Latency results reveal a different ranking. Hazelcast Jet records $128$1, $128$2, and $128$3. Flink records $128$4, $128$5, and $128$6 at $128$7, $128$8, and $128$9, respectively. Kafka Streams records $1024\,\mathrm{B}$0, $1024\,\mathrm{B}$1, and $1024\,\mathrm{B}$2. Spark exceeds $1024\,\mathrm{B}$3 at all three quantiles. The paper therefore concludes that Hazelcast Jet achieves sub-10 ms $1024\,\mathrm{B}$4 latency, that Flink and Kafka Streams are one to two orders of magnitude higher, and that Spark’s micro-batching enforces latencies in the seconds-to-minutes range.

The parameter study exposes several trade-offs. Raising maxOffsetsPerTrigger in Spark increases throughput to approximately $1024\,\mathrm{B}$5 at the cost of minute-long latency. Multi-core Pods produce negligible latency change and slight throughput gains for Flink and Hazelcast. Smaller records of $1024\,\mathrm{B}$6 improve throughput by up to $1024\,\mathrm{B}$7 for Hazelcast, while latency is unaffected. Reducing $1024\,\mathrm{B}$8 from $1024\,\mathrm{B}$9 to $P$0 raises Flink and Kafka Streams throughput by $P$1 but does not improve Hazelcast. Increasing $P$2 from $P$3 to $P$4 lowers throughput for all frameworks except Hazelcast, where the bottleneck lies elsewhere; the paper notes that latency may even drop because of buffering dynamics.

6. Interpretation, limitations, and prospective extensions

The paper’s overarching interpretation is that ShuffleBench isolates and stresses the shuffle+aggregate stage and thereby provides clear comparative insights into framework behavior under shuffle-heavy workloads. Within that framing, Flink is described as best-in-class for throughput, Hazelcast Jet for low latency, Kafka Streams as the middle ground, and Spark as limited by micro-batch overheads (Henning et al., 2024).

Its design also emphasizes reproducibility and external validity. The benchmark is open-source, uses Kubernetes orchestration, adopts default framework configurations, integrates with Theodolite, and relies on production-like deployments. The public artifact includes source code, Kubernetes manifests, and Theodolite experiment definitions at https://github.com/dynatrace-research/ShuffleBench. For researchers, this matters because it enables exact replication, controlled parameter sweeps, and extension to additional frameworks or alternative deployments.

The limitations and future directions are stated explicitly. One proposed extension is reliability and fault-tolerance evaluation, including recovery time and the impact of failures on throughput and latency. Another is semantic-aware shuffle optimizations, framed as the question of whether DSPFs can exploit overlaps among consumer predicates to reduce network I/O. Additional directions include non-uniform distributions such as skewed loads, heavy-tail consumer selectivities, and dynamic consumer lifecycles, as well as integration of SQL or other declarative layers to study the effect of higher-level APIs on shuffle-heavy workloads.

A plausible implication is that ShuffleBench occupies a distinctive position among DSPF benchmarks: it is not a general-purpose end-to-end analytics suite, but a focused instrument for workloads where routing semantics, keyed state locality, and redistribution cost dominate system behavior. In that sense, its significance lies less in defining a universal streaming benchmark than in making a previously underrepresented workload class measurable in a controlled and reproducible form.