Preparing for Performance Analysis at Exascale

Abstract: Performance tools for emerging heterogeneous exascale platforms must address two principal challenges when analyzing execution measurements. First, measurement of large-scale executions may record mountains of performance data. Second, performance measurements for parallel programs are sparse in two ways: the set of metrics present for any context and the set of contexts present in different threads. For GPU-accelerated applications, an important source of sparsity is that none of the myriad of GPU metrics apply to any of the many CPU contexts. To address these challenges, we developed a novel streaming aggregation approach to postmortem analysis that employs both shared and distributed memory parallelism to aggregate sparse performance measurements from every rank, thread, and GPU stream of an application, and attributes heterogeneous call path profiles and traces to source code. Using the same amount of resources, our approach analyzes large-scale performance measurements of GPU-accelerated applications over an order of magnitude faster than HPCToolkit and its sparse analysis results are as much as three orders of magnitude smaller than HPCToolkit's dense representation of metrics.