GainSight: Application-Guided Profiling for Composing Heterogeneous On-Chip Memories in AI Hardware Accelerators

Abstract: As AI workloads drive soaring memory requirements, higher-density on-chip memory is needed for domain-specific accelerators beyond what current SRAM technology can provide. We motivate that algorithms and application behavior should guide the composition of heterogeneous on-chip memories. However, little work has incorporated dynamic application profiles into these design decisions, and no existing tools are expressly designed for this purpose. We present GainSight, a profiling framework that analyzes fine-grained memory access patterns and data lifetimes in domain-specific accelerators. By instrumenting retargetable architectural simulator backends with application- and device-agnostic analytical frontends, GainSight aligns workload-specific traffic and lifetime metrics with mockups of emerging memory devices, informing system-level heterogeneous memory design. We also present a set of case studies on MLPerf Inference and PolyBench workloads using simulated GPU and systolic array architectures, highlighting the utility of GainSight and the insights it provides: (1) 64% of L1 and 18% of L2 GPU cache accesses, and 79% of systolic array scratchpad accesses across profiled workloads are short-lived and suitable for silicon-based gain cell RAM (Si-GCRAM); (2) Heterogeneous memory arrays that augment SRAM with GCRAM can reduce active energy consumption by up to 66.8%. To facilitate further research in this domain, GainSight is open source at https://gainsight.stanford.edu/.