--- title: Heimdall Benchmark Suite url: https://www.emergentmind.com/topics/heimdall-benchmark-suite type: topic --- # Heimdall Benchmark Suite Heimdall Benchmark Suite is an open-source, preemption-enabled suite designed for the evaluation of preemption strategies and scheduling policies in multi-tenant FPGA environments. It features 27 real-world workloads across cryptography, AI/ML, compute-intensive processing, communication systems, and multimedia domains. Each benchmark is equipped with integrated context-saving and restoration hooks to facilitate consistent and reproducible research in FPGA resource management and operating system design. Heimdall provides a standardized, extensible framework for methodical benchmarking, addressing the limitations of previous proprietary or synthetic approaches and enabling fair cross-comparison of scheduling algorithms [2511.06736]. ## 1. Design Motivations and Objectives The emergence of FPGAs as first-class cloud accelerators (e.g., AWS F1, Azure NP) has driven interest in multi-tenant operation, necessitating advanced support for preemptive multitasking. Vendor-native support for preemption is lacking, despite the growing demand for dynamic workload management. Preemption—defined as the sequence of pause, state-save, and resume operations—enables fine-grained time-multiplexing, superior utilization, and fairness in multi-tenant scenarios. Prior research on FPGA preemption and scheduling has suffered from methodological fragmentation, primarily due to the reliance on ad-hoc or proprietary benchmarks (such as STFS, Coyote, or ReconOS). This lack of a standardized, domain-spanning suite with context-save/restore support has impeded reproducibility and comparability. Heimdall’s core objectives are to: - Establish the first open-source, preemption-enabled FPGA benchmark suite. - Cover a broad spectrum of 27 workloads representative of cryptography, AI/ML, computational kernels, communication, and multimedia domains. - Integrate context-saving and restoration hooks into all designs for transparent preemption. - Support systematic, reproducible evaluation of resource management and scheduling strategies. ## 2. System Architecture and Orchestration ### 2.1 Partitioned FPGA Fabric Heimdall relies on Partial Reconfiguration (PR) to divide the programmable logic (PL) of an FPGA into distinct “slots,” each of which hosts a single accelerator. Slots can be sized heterogeneously according to the required resources (LUTs, flip-flops, BRAM, DSPs), enabling flexible mapping of different kernels. ### 2.2 Host-Side Scheduling Infrastructure A lightweight OS-like controller implemented on the ARM processing system (PS) of a Xilinx Zynq device leverages the Processor Configuration Access Port (PCAP) interface to: - Trigger accelerator suspension via clock gating and quiescence. - Read or write configuration frames to perform context capture and restoration. - Manage DRAM buffers that temporarily hold off-chip state during preemption events. Scheduling, including policy-based preemption (e.g., round-robin, priority queue), is enacted via this orchestration. The scheduling module determines which slots are suspended, saved, restored, and resumed according to configurable policies. ### 2.3 Context Save and Restore Implementation Context is preserved at the granularity of configuration frames, which are read from and written to the PL via the PCAP. For each accelerator slot: - On save: $N_{\text{frames}}$ frames are sequentially read into a contiguous region in DRAM. - On restore: The saved $N_{\text{frames}}$ are written back before resuming the design. - Hooks for context operations are automatically inserted into the RTL or processor workload; manual logic modifications are not needed. ## 3. Benchmark Suite Composition The 27 workloads in Heimdall are selected for real-world relevance, diversity of compute patterns, and variation in state size and latency. ### PL-Based Hardware Accelerators (15) - **AI/ML:** Convolution Layer (neural-network inference), Scalable Matrix Multiplication (dense GEMM) - **Multimedia/Image Processing:** PNG Decoder, JPEG Decoder, H.264 Video Encoder, Generic Image Processor (filter/color convert) - **Cryptography:** ML-KEM Server (NIST post-quantum key encapsulation), ML-KEM Client - **Signal & Math:** FFT Accelerator (radix-2), IIR Filter, FIR Filter, Trigonometry Core (CORDIC/polynomial) - **Communication/Interconnect:** Viterbi Decoder, Open NoC (network-on-chip router fabric) - **General Compute:** MIPS Soft Processor (bare-metal) ### RISC-V Processor-Based (12) - **Cryptographic Primitives:** AES-128 encryption, SHA-256 hashing, FALCON (lattice-based, with KeyGen, SignGen, SignVerify) - **MachSuite Data-Parallel Kernels:** BFS, Sort (quicksort), Needleman-Wunsch alignment, KMP search, dense GEMM - **CPU Performance:** Dhrystone, CoreMark | Category | Benchmarks (examples) | Characteristics | |---------------------|----------------------------------------|----------------------------------| | PL-based | Conv Layer, FFT, H.264 Encoder | RTL/HLS, varied state sizes | | RISC-V processor | AES-128, BFS, CoreMark | SW-centric, data-parallel, CPUs | ## 4. Context-Saving and Restoration Mechanisms Context management is quantitative and tunable per benchmark slot. Let $N_{\text{frames}}$ denote the number of configuration frames, $F_{\text{size}}$ the frame size in bits, and $B_{\text{bus}}$ the PCAP bandwidth. - **State size:** $S_{\text{state}} = N_{\text{frames}} \times F_{\text{size}}$ (bits) - **Save time:** $T_{\text{save}} = S_{\text{state}} / B_{\text{save}}$ - **Restore time:** $T_{\text{restore}} = S_{\text{state}} / B_{\text{restore}}$ - **Total preemption overhead:** $T_{\text{preempt}} = T_{\text{save}} + T_{\text{restore}}$ For the Xilinx XC7Z020: - $F_{\text{size}} = 101$ words $\times 32$ bits $= 3\,232$ bits/frame - $B_{\text{save}} \approx 3.2$ Gb/s (PCAP @ 100 MHz) - $T_{\text{save, per-frame}} \approx 1.01\ \mu$s - For $N_{\text{frames}} \approx 30$, $T_{\text{save}} \approx 30\ \mu$s$,\,T_{\text{restore}} \approx 32\ \mu$s$,\,T_{\text{preempt}} \approx 62\ \mu$s$ This strict methodology for state management facilitates reproducible measurement of preemption costs across domains. ## 5. Scheduling and Evaluation Metrics Heimdall standardizes the measurement of preemption and scheduling through canonical metrics: - **Fairness Index (Jain's):** $F = \frac{(\sum_i x_i)^2}{n \sum_i x_i^2}$, where $x_i$ is the throughput or job count for tenant $i$, $n$ is the number of tenants. - **Resource Utilization:** $U = \frac{\sum_t \text{active\_time}_t}{T_{\text{total}}}$, averaged over all $t$. - **Scheduling Latency:** $L_{\text{sched}}$ is the mean interval from preemption request to quiesce/restore. - **Turnaround Time:** $T_{\text{turn}} = \text{waiting time} + T_{\text{execution}} + (\text{number of preempts} \times T_{\text{preempt}})$ All metrics are evaluated over a scheduling epoch of duration $T_{\text{total}}$. This consistent framework permits apples-to-apples comparison of varied scheduling (e.g., fairness, utilization, latency) and preemption protocols. ## 6. Benchmarking Setup and Workflow ### Hardware Platform - Xilinx Zynq-7000 SoC (XC7Z020, ARM Cortex-A9 with PL fabric) - Vivado 2018.2 toolchain; PCAP interface at 100 MHz ### Software Stack - C-based PS-side orchestration using Xilinx SDK 2018.2 - PCAP driver for configuration access - Scheduling daemon with configurable quantum and policies ### Experimental Procedure 1. Synthesize each benchmark with preemption hooks. 2. Partition PL into the desired number of slots; map accelerators via PR. 3. For each experimental run: - Trigger state save; record timestamps for $T_{\text{save}}$. - Trigger state restore; record for $T_{\text{restore}}$. - Log throughput, latency, and energy measurements (if available). 4. Post-process data to compute $F$, $U$, $L_{\text{sched}}$, $T_{\text{preempt}}$, and $T_{\text{turn}}$ for each policy. ## 7. Extensibility and Guidelines for Adding New Benchmarks Researchers can extend Heimdall as follows: 1. Define a new PR region in Vivado (area, interface, clock). 2. Adapt the RTL/HLS kernel with pause/resume hooks for all stateful logic (FFs, BRAMs, DSP regs) and implement frame-read/write loops per PS control. 3. Specify $N_{\text{frames}}$, $F_{\text{size}}$, and DRAM address for $S_{\text{state}}$ in the scheduler. 4. Develop a functional test that exercises pause–resume at random intervals and checks for bit-identical output. 5. Contribute the following to the repository: - Source code (RTL/HLS), block design, preemption wrapper - PS control code snippet for save/restore invocation - Documentation (README) detailing resource usage, latency, and $S_{\text{state}}$ This rigorous process ensures all contributed benchmarks adhere to Heimdall’s reproducibility and preemption standards, supporting robust, comparative research in multi-tenant FPGA management [2511.06736]. Source: https://www.emergentmind.com/topics/heimdall-benchmark-suite