StreamBox-HBM: Hybrid Memory Stream Analytics

• StreamBox-HBM is a stream analytics engine that combines DDR4 for capacity with 3D-stacked HBM for high-bandwidth processing.
• It uses innovative key pointer arrays and vectorized sort/merge algorithms to efficiently handle grouping operations in streaming data.
• Dynamic runtime memory management and parallel processing on multi-core systems enable throughput of up to 110 million records per second.

StreamBox-HBM is a modern stream analytics engine designed to exploit heterogeneous hybrid memory architectures combining commodity DDR4 DRAM and 3D-stacked High Bandwidth Memory (HBM). Its primary innovation lies in reconciling the stringent memory bandwidth requirements of streaming data analytics—typically involving random access, high capacity, and rapid grouping operations—with the architectural strengths and weaknesses of HBM (limited capacity, maximal sequential bandwidth, and parallel access optimization). Contrary to the intuition that streaming analytics are a poor match for HBM due to capacity and access pattern constraints, StreamBox-HBM achieves scalable, high performance by introducing algorithmic and architectural strategies tailored to the memory system. Notably, the system demonstrates substantial throughput advantages—processing up to 110 million records per second and saturating HBM bandwidth at up to 238 GB/s—while utilizing all 64 cores of an Intel Knights Landing (KNL) system. Its grouping operations outperform alternative algorithms by factors of 7× (over sequential algorithms without its custom structures) and by an order of magnitude over conventional random-access hashing approaches (Miao et al., 2019).

1. Hybrid Memory System Architecture

StreamBox-HBM operates on platforms that offer both standard DRAM (for capacity) and HBM (for bandwidth). Incoming streams are first materialized as full records into DRAM, which acts as a high-capacity repository. When stream operators require grouping, StreamBox-HBM dynamically extracts Key Pointer Array (KPA) data structures: compact sequences of keys and pointers referencing the relevant records stored in DRAM. Only these KPAs—containing the minimum necessary information—are allocated in HBM, enabling grouping computations to be performed using sequential access and vectorized instructions.

Architectural flow:

• Ingestion: Full records arrive via RDMA or network stack, stored in DRAM row format.
• Extraction: For each grouping, the engine builds a KPA with (key, pointer) pairs in HBM.
• Computation: KPAs are grouped using parallel, vectorized sort/merge routines entirely inside HBM.
• Resource Monitoring: The runtime continuously measures HBM capacity and DRAM bandwidth, dynamically steering further allocations to maintain resource balance.

This architecture capitalizes on HBM's sequential access and high concurrency potential while exploiting DRAM for high volume persistent storage.

2. Sequential Data Grouping Algorithms and Key Pointer Arrays (KPA)

Conventional stream analytics engines use hash-based grouping (requiring random access over large working sets in DRAM), which mismatches HBM’s optimal sequential access model. StreamBox-HBM departs from this by:

• Sorting-Based Grouping: KPAs are sorted using chunked, parallel merge-sort algorithms, implemented with AVX-512–tuned vector kernels.
• Join/Merge on KPAs: Join operations leverage sequential scan merge, eliminating random access overhead.
• KPA Data Model: KPAs separate “hot” columns (resident key) in HBM from “cold” payload data in DRAM. Key swaps allow fast adaptation as grouping keys change; reference counts on DRAM bundles ensure storage reclamation without moving full records.

Together, these mechanisms allow grouping and aggregation over massive streams, taking full advantage of HBM’s bandwidth for critical intermediate operations while sidestepping capacity limitations.

3. Dynamic Runtime Memory and Resource Allocation

A "demand balance knob" regulates allocation of KPAs between HBM and DRAM, with scalar parameters $k_{\text{low}}$ and $k_{\text{high}}$ controlling probabilistic placement:

$$\text{if (allocation\_performance\_tag == Urgent)} \;\; M = \text{HBM}$$

$$\text{else if (allocation\_performance\_tag == High)} \;\; M = \begin{cases} \text{HBM} & \text{if random(0,1) < } k_{\text{high}} \ \text{DRAM} & \text{otherwise} \end{cases}$$

$$\text{else if (allocation\_performance\_tag == Low)} \;\; M = \begin{cases} \text{HBM} & \text{if random(0,1) < } k_{\text{low}} \ \text{DRAM} & \text{otherwise} \end{cases}$$

The parameters are refreshed every 10 ms based on direct telemetry. Urgent, latency-critical pipeline KPAs always allocate in HBM; others are assigned probabilistically to maintain overall system health and ensure neither memory resource becomes a bottleneck.

4. Performance Metrics and Scalability

Empirical benchmarks on a 64-core KNL node demonstrate:

Metric	StreamBox-HBM	Sequential w/o KPA	Random-access Hashing
Throughput	110 M records/s	up to 7× slower	>10× slower
Memory Bandwidth	150–250 GB/s (64 cores)	<80 GB/s	<20 GB/s
Core Efficiency	18× higher per core	Reference	Reference

Overall, the system sustains up to 70% of HBM’s theoretical peak bandwidth. Throughput scales linearly with core count until HBM capacity or DRAM bandwidth imbalance intervenes.

5. Technical Challenges and Solutions

Three primary technical challenges addressed by StreamBox-HBM:

• HBM Capacity Limitation: KPA extraction and key/value separation minimize HBM footprint, ensuring only hot, intermediate data occupies precious bandwidth region.
• Algorithm–Memory Mismatch: Replacement of random-access hash grouping with sequential AVX-512–vectorized sort/merge routines exploits HBM's layout.
• Dynamic Memory Management: Real-time adjustment of KPA allocation—driven by application context—maintains a balance between DRAM's capacity and HBM's bandwidth, eliminating resource stalls and collisions.

Adjacent processes (materialization, extraction) are coalesced to reduce data movement. KPAs are never migrated; they are instantiated afresh based on pipeline stage requirements and resource status.

6. Integration with Broader Accelerator and Database Ecosystems

StreamBox-HBM’s approach is directly extensible to high-bandwidth memory accelerator design in FPGAs and hybrid high-performance computing environments (Kara et al., 2020). Key lessons translate:

• Database Integration: By allocating only indexing and grouping intermediates in HBM, the system achieves substantial query acceleration in database engines, with up to 12.9× speedup for join operations and over 3.2× for SGD workloads.
• Data Partitioning and Channel Mapping: To exploit full HBM bandwidth, data streams are statically and dynamically mapped to independent HBM channels, as in MoneyDB accelerator integration. Efficient partitioning and floorplanning minimize crossbar and resource congestion.

The architecture is compatible with emerging database acceleration frameworks and theoretical improvements such as OpenCAPI-integrated memory access overlays.

7. Practical Optimization Strategies

StreamBox-HBM’s deployment yields several general principles applicable to hybrid memory systems:

• Access Pattern Profiling: Fine-grained counters aid in detecting workload phases optimal for HBM—regular, sequential accesses are prioritized.
• Adaptive Hot Data Placement: Real-time migration or replication of hot data into HBM maximizes pipeline throughput, especially for time-varying streaming workloads.
• Concurrency Tuning: Thread and pipeline parallelism are calibrated to saturate HBM bandwidth, avoiding contention but maximizing concurrency.
• Memory Configuration Selection: Both flat and cache mode are supported, with empirical tuning based on workload characteristics and resource saturation telemetry.

Periodic dynamic adjustment ensures that scaling behavior persists across variable ingest rates and query complexities.

---

StreamBox-HBM represents a sophisticated solution to the challenge of maximizing hybrid memory system throughput for demanding real-time stream analytics. By blending algorithmic, architectural, and resource management innovations, the system unlocks the practical performance gains latent in high-bandwidth memory technologies, particularly for streaming workloads traditionally considered a poor fit for HBM. Its influence extends to accelerator integration in databases and FPGAs, offering a blueprint for next-generation high-throughput, low-latency data analytics systems on heterogeneous memory platforms.