Streamo-Bench: Modular Streaming Benchmark

• Streamo-Bench is a modular benchmarking platform that evaluates streaming-sampling algorithms under realistic conditions with concept drifts and micro-bursts.
• It employs synthetic data streams generated through configurable stochastic models, ensuring reproducibility and realistic simulation of network artefacts.
• Its layered architecture, including ranking engines and hyperparameter optimizers, standardizes metric collection and supports extensible integration of diverse algorithms.

Streamo-Bench is a modular, extensible benchmarking platform developed to evaluate and rank streaming-sampling algorithms under realistic, dynamic data stream scenarios. Its architecture provides a configurable environment for generating synthetic streams with concept drifts and micro-bursts, supports diverse algorithm families via a standardized API, and collects detailed, online metrics to assess both estimation accuracy and downstream partitioning impact. By publishing all modules and protocols, Streamo-Bench enables reproducible, extensible experimental evaluation for researchers focused on distribution estimation over data streams (Gáspár et al., 2020).

1. System Architecture and Component Interfaces

Streamo-Bench comprises five principal modules:

1. Data Generator: Produces synthetic streams with parameterized concept drifts and micro-burst events, governed by a metadata configuration file including stream length, key universe size, micro-batch size, drift/burst parameters, and random seeds.
2. Sampling/Benchmarking Core: Executes each candidate streaming-sampling algorithm in isolation, feeding the identical data stream in micro-batches.
3. Ranking Engine: Aggregates per-batch output and computes multiple accuracy and impact metrics—such as Hellinger distance, KL-divergence, MSE, and simulated partition load imbalance—across all tested algorithms.
4. Hyperparameter Optimizer: Operates in closed loop with the Ranking Engine, exploring hyperparameter space via random local search using aggregate metric scores as the fitness signal.
5. Orchestration & Metadata Manager: Drives experiment execution by reading configuration files, managing parallelism, storing results, and handling module interaction through clean, versioned APIs.

This layered architecture, with well-defined interfaces (stream + metadata generator API, SamplerBase for per-key processing, metrics collector at batch boundaries, ranking API, hyperparameter optimizer callback), accommodates extensibility and strict reproducibility across experimental runs (Gáspár et al., 2020).

2. Synthetic Data Stream Generation Methods

The Data Generator module supports advanced stochastic modeling of data streams:

• Concept Drift Modeling: Drifts are encoded as tuples $d=(\ell,\mathit{mid},P_1,P_2)$, where $\ell$ is the drift length, $\mathit{mid}$ is its center, and $P_1,P_2$ are start/end discrete distributions. For gradual drifts ($\ell>0$), batch distributions mix $P_1$ and $P_2$ with probability $p_i$, realizing a randomized linear interpolation; abrupt drifts ($\ell=0$) switch distributions instantaneously.
• Micro-Burst Simulation: Overlaid bursts introduce transient “router stall” effects. Bursts can be configured by start probability, “held-back” key probability, and active duration interval $[\Delta_{\min},\Delta_{\max}]$. Held-back keys accumulate in a map and are flushed collectively at burst end. All randomness is seeded for experiment determinism.

These mechanisms enable reproducible investigation of both distributional shifts and real-world network-induced artefacts (Gáspár et al., 2020).

3. Supported Algorithms and Extensibility

Any algorithm conforming to the SamplerBase interface (key-by-key ingestion, on-demand estimation, memory reporting) can be benchmarked. Streamo-Bench provides:

• Counter-Based Techniques: Sticky Sampling, Lossy Counting [Manku, 2002], SpaceSaving [Metwally et al.], Frequent [Golab & Graham, 2003]
• Sketch-Based Techniques: Count-Min Sketch [Cormode & Muthukrishnan, 2005], Count Sketch [Charikar et al., 2002]
• Change-Respecting: Landmark [Zhu & Shasha, 2002]
• Drift-Adaptive Algorithms:
  • Temporal Smoothed: maintains main and secondary samplers, swaps based on scheduled intervals for rapid post-drift adaptation.
  • Checkpoint Smoothed: spawns parallel samplers at fixed checkpoints, uses Hellinger distance to trigger main sampler replacement when distribution shifts are detected.

To extend the framework, new samplers are plugged in by implementing the interface, adding metadata entries, and recompiling. New data generators and optimizers are similarly integrated via subclassing and interface compliance (Gáspár et al., 2020).

4. Evaluation, Metrics, and Ranking Methodologies

Streamo-Bench employs two primary ranking paradigms:

• Direct Distribution Estimation Metrics:
  • Hellinger distance $\ell$0 between true and estimated batch distributions, measured per micro-batch, aggregated via mean, max, or percentile.
  • Kullback–Leibler divergence (with smoothing for support mismatches) and mean squared error for top-$\ell$1 frequency estimates.
  • Composite scores combine multiple metrics with user-specified weights.
• Downstream Partition Load Metrics:
  • A parallel simulation estimates partition loads after hash-based grouping, driven by the estimated key frequencies.
  • Percent imbalance $\ell$2 quantifies distribution-induced load skew.

Rankings for each algorithm can be metric-specific or summarized in a multidimensional scoreboard (Gáspár et al., 2020).

5. Experimental Protocols and Reproducibility Guarantees

Reproducibility is emphasized via fully specified experiment descriptors:

• Default settings: stream length $\ell$3M, $\ell$4, micro-batch size $\ell$5, top-$\ell$6.
• Drift scenarios: one midpoint gradual ($\ell$7) and abrupt ($\ell$8) shift between Zipfian distributions.
• Burst configurations: none, “light” ($\ell$9), “heavy” ($\mathit{mid}$0).
• Partition simulation uses $\mathit{mid}$1, with all resources and random seeds logged.

Experiments executed on a standardized hardware baseline (6-core/12-thread AMD Ryzen 3600, 16GB RAM) ensure comparability; all definitions, logs, and results are public (Gáspár et al., 2020).

6. Empirical Findings and Comparative Analysis

Quantitative results reveal:

Algorithm	Recovery (post-gradual drift)	Suppression (burst noise)	Memory	Runtime (s)
Frequent	Immediate	Larger transient spikes	Baseline	1.35/1.17
Lossy Counting	5–10 batches lag	Larger transient spikes	$\mathit{mid}$2	1.49/1.39
Temporal Smoothed	Immediate	Some spikes	$\mathit{mid}$3 Frequent	1.98/1.78
Checkpoint Smoothed	5–6 batches lag	Peak $\mathit{mid}$4	$\mathit{mid}$5 +30%	2.09/1.85

• Abrupt drifts induce spikes in load imbalance ($\mathit{mid}$6); Temporal Smoothed reduces to $\mathit{mid}$7 in 3–4 batches, outperforming Checkpoint Smoothed (5–6 batches) and Lossy Counting ($\mathit{mid}$8 batches).
• Checkpoint Smoothed best suppresses burst-induced noise in estimation.
• Memory and computation cost for drift-adaptive variants are moderate relative to base samplers (Gáspár et al., 2020).

7. Implementation Details and Community Accessibility

Streamo-Bench is implemented in Java 11 with a modular plugin system. Experiments are YAML/JSON-driven and validated by an explicit schema. Extending the framework requires implementing defined interfaces and minimal metadata configuration. All source, runners, and experiment logs, including seeds for deterministic reproduction, are available in the official repository.

This reproducibility and openness positions Streamo-Bench as a foundational platform for systematic benchmarking and incremental research in data stream analytics and distributed online sampling (Gáspár et al., 2020).