HyDRA: Deadline and Reuse-Aware Cacheability for Hardware Accelerators

Abstract: The system-level cache is a critical resource shared by processor cores and domain-specific accelerators in heterogeneous systems on chips (SoCs). The strict QoS requirements of accelerators, such as deadlines, can lead to severe performance degradation of processor cores. Thus, managing the shared cache efficiently between cores and accelerators becomes crucial. State-of-the-art cache management techniques perform reuse-aware bypassing of accesses from cores with the help of reuse predictors to improve performance. However, architectural differences between accelerators and processor cores (often associated with deep cache hierarchies) can lead to significantly different reuse patterns at the shared cache. We propose a novel clustering-based methodology, LERN, for learning and predicting the reuse behavior of hardware accelerators at the shared cache. We then propose a deadline and reuse-aware cache management strategy, HyDRA, which explores a novel tradeoff between reuse and deadline awareness for performance efficiency. It uses LERN to dynamically predict the reuse behavior of the accelerator accesses and make bypass decisions to maximize the system throughput while meeting accelerator deadlines. We evaluate HyDRA across different workloads and varied accelerator configurations. It significantly improves the system performance and reduces the accelerator deadline miss rate.

• The paper introduces HyDRA, a cache management strategy that dynamically adapts bypass policies based on accelerator progress and reuse predictions, achieving up to 90% reduction in deadline miss rate.
• The paper presents LERN, a clustering-based reuse predictor that analyzes reuse intervals and counts to accurately classify accelerator memory accesses with up to 100% cluster assignment accuracy.
• The paper validates its approach across diverse ML workloads and system configurations, demonstrating improvements in IPC, robust deadline handling, and minimal hardware overhead compared to prior methods.

HyDRA: Deadline- and Reuse-Aware Cacheability for Hardware Accelerators

Motivation and Problem Statement

Integrating domain-specific hardware accelerators (HWAs) within multicore System-on-Chips (SoCs) is a defining trend for edge and AI-capable systems. These accelerators share critical system resources such as the last-level cache (LLC) with general-purpose processor cores, introducing complex contention in both space and bandwidth. Unlike traditional CPU-GPU workloads, HWAs exhibit distinct memory access and reuse characteristics, often lacking program counter (PC)-based features and demonstrating memory patterns with long-range correlations or irregular reuse. Most prior cache management works focus on utilization, cache partitioning, and reuse-aware bypass for cores or GPUs, but do not address HWAs’ strict quality-of-service (QoS) requirements such as completion deadlines.

Existing reuse predictors and cache bypass strategies either ignore accelerator deadline constraints or fail to account for the architectural and application-specific differences in HWA memory access patterns at the LLC. These mismatches can significantly degrade system-level performance metrics—namely, core throughput (IPC) and deadline miss rate (DMR) for HWAs. Thus, there is a need for cache management mechanisms that reconcile the requirements of both cores and accelerators, modulating cache allocation decisions based on both predicted reuse and real-time deadline awareness.

LERN: Clustering-Based Reuse Prediction for Accelerators

The fundamental technical innovation in this work is the LERN framework for reuse analysis and prediction tailored to accelerator traffic at the shared cache. Unlike PC-based, online-trained predictors (e.g., SHIP), LERN statically profiles accelerator traces, extracting memory access signatures in terms of reuse interval (RI) and reuse count (RC).

This method builds a two-dimensional vector space where each cache line address is characterized by:

• Reuse Interval (RI): Distribution across bins reflecting cache associativity and anticipated eviction probabilities.
• Reuse Count (RC): Total accesses per line address within the profiled sequence.

LERN employs K-means clustering on these features, segmenting accesses into semantically meaningful categories (e.g., Cold/Light/Moderate/Hot in RC; Immediate/Near/Far/Remote in RI). The resulting clusters provide coarse-grained but substantially more discriminative predictions than conventional binary live/dead clusterings. LERN thus directly learns reuse patterns from accelerator traces, achieves up to 100% cluster assignment accuracy in favorable configurations, and exposes the heterogeneity of accelerator memory behavior not captured by prior solutions.

HyDRA: Joint Deadline- and Reuse-Aware Shared Cache Management

HyDRA builds on LERN by introducing a distributed, adaptive cache bypass policy with two core properties:

1. Deadline Awareness: HyDRA divides execution into epochs, dynamically monitors accelerator progress (completed accesses vs. target rate derived from deadline), and adjusts bypass aggressiveness. Dedicated progress monitors and safety margins compensate for unpredictable access rate bursts and lower-level memory contention.
2. Reuse Awareness: Each accelerator access is classified using LERN’s offline-trained cluster tables, deployed as a direct-mapped, tagless hardware table (L-RPT). Bypass decisions are then formulated as thresholded queries against the current RI and RC cluster annotation, merging knowledge of likelihood of reuse and proximity to deadline miss.

If accelerator progress exceeds the required schedule, HyDRA aggressively bypasses even accesses considered moderately reusable, reallocating cache space/bandwidth to cores. If progress lags, bypass is throttled to prioritize cache residency of potentially reusable accelerator lines, minimizing DMR. This explicit tradeoff results in a genuinely dynamic, application- and system-sensitive policy.

Cache bypass for cores is performed using any state-of-the-art predictor (SHIP by default), further boosting IPC under accelerator-favorable bypass regimes.

Experimental Results

HyDRA is implemented and exhaustively evaluated using an enhanced version of gem5 with integrated accelerator models and a wide range of edge-representative system parameters. Key results include:

• IPC and DMR Tradeoff: On standard edge SoC topologies (eight OOO cores, cache sizes from 1MB–16MB, realistic memory hierarchies), HyDRA consistently delivers 2–11% core IPC improvement and up to 90% reduction in accelerator DMR compared to both default (FIFO-NB) and deadline-aware SHIP-based baselines.
• Reuse Predictor Efficacy: LERN-driven bypass achieves nontrivial accelerator bypass (up to 10%) even at high LLC hit rates, where PC-based SHIP fails to discriminate accelerator traffic.
• Deadline Robustness: Deadline-miss rates are strictly bounded under all cache and accelerator configurations, even when aggressive bypass is required for IPC recovery.
• Policy Generality: HyDRA’s gains generalize to a broad suite of ML workloads (Tiny-YOLO, GoogleNet, MobileNet, Faster R-CNN, AlphaGoZero, Deep Speech2) and diverse accelerator microarchitectures. Scenario evaluations with architectural variations (larger L2, cache prefetchers, way partitioning, alternative memory technologies) confirm robustness.
• Hardware Overhead: LERN predictor tables require 3.9% additional cache controller area (7.1% total for the full HyDRA logic), with direct-mapped one-cycle lookups and negligible scheduler critical-path interference. Energy overhead is bounded within ±8% of baseline.
• Comparison With Prior Work: State-of-the-art cache partitioning and prefetching methods such as DPCP [DPCP], FLASH [FLASH], or static way partitioning underperform HyDRA both in IPC and DMR, often failing to uphold deadlines or over-constraining core access to the LLC.

Implications and Future Directions

HyDRA validates the necessity of decoupling accelerator cache management from general-purpose predictors, and explicitly incorporating both deadline knowledge and domain-specific reuse models. The success of LERN underscores the advantage of hybrid offline training and simple online hardware deployment, in contrast to recent deep learning predictors, which remain hardware-prohibitive for on-chip deployment. It also demonstrates the non-trivial interplay between bypass granularity, access pattern heterogeneity, and system-level performance objectives—invalidating the one-size-fits-all approaches common in MPSoC resource management.

Practically, HyDRA’s methodology is applicable to modern and future SoC designs with ML/NPU blocks and highly variable memory traffic, including NXP LS2088A-style non-sliced LLCs, and could be adapted per slice in cluster-distributed caches. The flexibility in thresholding, clustering depth, and architectural parameterization permits deployment across edge (mobile, automotive, IoT) and server-grade AI systems.

Theoretically, this work highlights the open research axis in joint bandwidth and space management, SLO-driven cache partitioning, and multi-objective control (e.g., performance, energy, and endurance for NVM caches). Extending the paradigm to workloads with highly non-stationary data locality (e.g., GNNs), modeling L4 and off-chip cache interactions, and integrating runtime learning for unforeseen dynamic reuse patterns will be important next steps.

Conclusion

HyDRA presents a comprehensive deadline- and reuse-aware cache bypass and allocation strategy for multicore/accelerator SoCs. By decomposing accelerator memory access behavior and integrating progress-driven control over bypass aggressiveness, HyDRA consistently outperforms both traditional and deadline-aware baselines—with robust improvements in IPC and deadline conformance. The methodology, enabled by the LERN clustering framework, fills a critical gap in heterogeneous cache management and sets a precedent for future SoC hardware and systems research (2605.08908).