---
title: 'hls4ml: Hardware-Aware ML Compiler'
url: https://www.emergentmind.com/topics/hls4ml
type: topic
---

# hls4ml: Hardware-Aware ML Compiler

hls4ml is an open-source software platform and hardware generation framework that automates the transformation of trained machine learning (ML) models into high-level synthesis (HLS) code suitable for deployment on field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs). Architecturally, hls4ml operates as a modular “compiler,” ingesting models from Keras (including QKeras), PyTorch, and ONNX/QONNX, applying hardware-aware optimizations, and emitting hardware-amenable C++ or SystemC ready for vendor HLS tools such as Vivado HLS, Intel oneAPI DPC++, and Catapult HLS. Its core strengths lie in supporting quantized and pruned neural networks, streaming dataflow architectures, sub-microsecond inference latencies, and the ability to target diverse platforms, making it a key tool for domains demanding extreme resource efficiency and real-time inference—most notably scientific instrumentation and edge computing applications [2512.01463][2103.05579].

## 1. System Architecture and Software Workflow

hls4ml is architected around a multi-stage compilation flow: front ends for parsing ML models, a sequence of hardware-oriented optimizations, and back ends for code generation targeting major HLS toolchains. The frontend parsers support Keras (all TensorFlow versions, QKeras quantizers), PyTorch (FX, Brevitas, HAWQ), and ONNX/QONNX, extracting weights, quantization parameters, graph topology, and layer configuration into a device-agnostic internal representation (“ModelGraph”). 

Subsequent optimizer flows perform transformations such as affine+BatchNorm fusion, precision propagation, channel format transpositions, per-layer bitwidth inference, constant folding, streaming FIFO depth minimization (via RTL-level occupancy profiling), and the splitting of graphs for multi-component hardware synthesis. User configuration is provided as Python dictionaries or YAML files, setting parameters such as per-layer precision, reuse factors, dataflow vs. parallel I/O, distribution of resources, and synthesis strategies (‘Latency’, ‘Resource’, or ‘Distributed Arithmetic’).

The backend maps the IR to hand-optimized C++/SystemC templates for each supported layer, emits HLS pragmas (e.g., `#pragma HLS dataflow`, `#pragma HLS pipeline II=1`, `#pragma HLS array_partition`), generates vendor toolchain scripts, and writes test benches for simulation/cosimulation. hls4ml supports direct flow for FPGAs via Vivado/Vitis (Xilinx), Intel oneAPI/Quartus, and Catapult HLS (Siemens) for both FPGA and hierarchical ASIC flows [2512.01463][2103.05579][2103.13060].

## 2. Hardware Mapping Strategies: Dataflow, Pipelining, and Quantization

The hardware designs produced by hls4ml are deeply pipelined, dataflow-oriented accelerators that maintain all network parameters and activations in on-chip memory—completely obviating DRAM access in real-time applications. Each major operator (Dense, Conv2D, GRU, LSTM, MultiHeadAttention, Pooling, BatchNorm, etc.) is instantiated as an HLS module, connected via streaming interfaces (e.g., `hls::stream`). Layer-to-layer streaming and pipelining are orchestrated with HLS dataflow pragmas to parallelize both intra- and inter-layer computation.

Precision across weights, activations, and accumulators is flexibly controlled at the per-layer level through fixed-point (`ap_fixed<W,I>`), binary, ternary, or even custom floating-point datatypes via QKeras, HGQ, or ONNX/QONNX quantizer metadata [2512.01463][2003.06308][2304.06745]. hls4ml supports explicit configuration of accumulator widths using analytical formulas incorporating input/output shapes and quantizer bit-widths to guarantee no overflow during MAC operations.

Key backend strategies include:
- **Latency strategy:** maximum parallelization/unrolling (Reuse factor RF=1, II=1), minimized inference latency, maximal instantaneous resource usage [2512.01463].
- **Resource strategy:** increased reuse factor, time-multiplexed multiplication pipeline, II>1, reduced DSP/LUT/BRAM consumption.
- **Distributed Arithmetic (DA):** via the integrated da4ml algorithm, replacing constant-matrix–vector multiplies with shift-and-add LUT logic, trading off DSPs for LUTs and enabling resource reduction up to 30% for highly quantized models [2507.04535].

## 3. Neural Network Types and Supported Model Classes

hls4ml supports a broad spectrum of neural architectures:
- **Dense/MLP:** Fully connected networks with per-layer quantization and pruning—binary/ternary and hybrid quantization are natively supported with automatic resource elimination of DSPs when possible [2003.06308].
- **Convolutional Neural Networks (CNNs):** Streaming 1D/2D/3D convolutions using hardware-efficient shift-register line buffers, supporting compressed and quantized kernels with automated BRAM/LUT balancing; see semantic segmentation with ENet on ZCU102 at sub-5 ms latencies [2205.07690][2101.05108].
- **Recurrent Neural Networks:** LSTM and GRU with user-controlled static/non-static modes and reuse factors, enabling latency-resource trade-off and achieving <2 μs inference for jet tagging and other scientific workloads [2207.00559].
- **Graph Neural Networks:** Automated lowering of Keras/ONNX GNN topologies, including message-passing–style interaction networks for charged particle track reconstruction (<1 μs latency on Kintex UltraScale) [2012.01563].
- **Transformer Architectures:** Multi-head attention, softmax, and (optionally) layer normalization, with specialized LUT-based implementations for softmax and exp/inverse, enabling sub-2 μs inference on UltraScale+; arbitrary Keras/ONNX transformers can be mapped by adjusting per-layer configuration [2402.01047][2409.05207].
- **Boosted Decision Trees:** Direct support for BDTs from scikit-learn/XGBoost/TMVA, instantiated as parallel combinational logic trees, with <100 ns latency for Level-1 triggers [2002.02534].

## 4. Model Compression, Quantization, and Design-Space Exploration

hls4ml is tightly coupled with quantization- and pruning-aware training workflows:
- **QAT/PTQ:** Integration with QKeras, HGQ, Brevitas, and ONNX/QONNX allows both post-training and quantization-aware training, including asymmetric, uniform, heterogeneous, or Hessian-aware mixed-precision schemes [2304.06745][2512.01463][2003.06308].
- **Pruning:** Model-optimizer passes support structured and unstructured sparsity. Parameters with magnitude below threshold are pruned and corresponding hardware computation is dropped [2103.05579][2501.05515].
- **Automated Design Sweeps:** Python/YAML configuration variables allow global and per-layer sweeps of bit-width, reuse-factor (“parallelism” vs. “latency/resource”), precision, and resourcing strategies.
- **Surrogate Modeling:** wa-hls4ml and rule4ml provide neural and regression surrogate models for instantaneous prediction of LUT, DSP, FF, BRAM, and latency, trained on >600,000 hls4ml synthesized networks, accurately predicting cycle counts and resources within 10–30% for prospective designs, enabling rapid architecture-hardware co-design [2511.05615][2408.05314].

| Surrogate Tool | Type      | Metrics Predicted         | R² Range | sMAPE Range        |
|----------------|-----------|--------------------------|----------|--------------------|
| wa-hls4ml      | GNN/Trans | LUT, DSP, FF, BRAM, Cyc. | 0.89–0.95| 2.9–15.7% (test set)|
| rule4ml        | MLP       | LUT, DSP, FF, BRAM, Cyc. | 0.8–0.98 | 10–30%             |

## 5. Device Backends, Performance, and Supported Applications

hls4ml supports FPGAs (Xilinx: Vivado HLS, Vitis HLS; Intel: oneAPI/Quartus HLS), and ASIC flows (Siemens Catapult HLS). Each backend supplies tailored HLS templates for dataflow-pipelined inference, with interface generators for both parallel and streaming protocols (AXI4-Stream, OneAPI pipes, custom memory-mapped buffers). By leveraging platform-specific pragmas (automatic in backends) and portable precision types (ap_fixed, ac_fixed), device support is extensible (e.g., Alveo, Zynq, UltraScale+, Arria/Agilex, and ASICs) [2512.01463][2103.13060].

Reported performance metrics include:
- **Resource-efficiency:** Dense MLPs (5–100k parameters) at II=1 with 0–1% of chip DSPs via full latency-unrolling and distributed arithmetic (DA) [2507.04535][2512.01463].
- **Ultra-low latency:** Sub-2 μs CNNs/Transformers; 10–100 ns for small MLPs, and <1 ms end-to-end for complex sensor-fusion architectures such as U-Net on Arria 10 [2101.05108][2409.05207][2311.05716].
- **Energy/Throughput:** O(10 ns) latency, 200 M inference/s for small MLPs; O(1 μJ) per inference on FPGA, nanosecond-range and nJ-scale on 28–65 nm ASICs [2103.05579][2512.01463].

hls4ml has enabled a variety of scientific deployments:
- **Trigger-level physics ML:** Level-1 collider triggers (jet tagging, muon momentum, anomaly detection) at 40 MHz, sub-100 ns budget [2002.02534][2311.05716][2304.06745].
- **Edge control and sensor fusion:** Real-time beam-loss tracking on Arria 10 (575 fps, 1.7 ms), multi-channel qubit readout (32 ns latency) [2311.05716][2501.14663].
- **Autonomous perception:** Real-time multi-class semantic segmentation, e.g., ENet on ZCU102 at 4.8 ms/image, 3 W dynamic power [2205.07690].

## 6. Extensibility, Portability, and Research Ecosystem

hls4ml is designed for extensibility. New layer types, quantizers, and optimization passes can be added via an Extension API at the IR and backend template level [2512.01463]. Conversion routines and resource estimation flows are continually developed alongside higher-level ecosystem tools, such as wa-hls4ml (benchmark and surrogate prediction), rule4ml (fast regression predictors), da4ml (distribution arithmetic), and experimental improvements in backend portability (constexpr-based LUTs, custom floating-point support, ac_fixed/ac_int, and multi-vendor pragma abstraction) [2507.04535][2103.13060][2511.05615][2408.05314].

hls4ml has demonstrated robust support for a wide range of real-world ML workloads under severe hardware and latency constraints, facilitating the deployment of compressed, quantized, and low-power ML on reconfigurable and custom hardware in scientific and industrial contexts. Its contributions are documented in open-source repositories and benchmarked across contemporary accelerator toolchains [2512.01463][2103.05579][2511.05615].

## 7. Limitations, Ongoing Development, and Outlook

Key limitations historically identified include backend specialization (Vivado-only pragmas, Xilinx-specific types), limited support for custom FP types, and code size/memory for large models. Recent work addresses these limitations by “de-specializing” the codebase, introducing portable LUT generators, ac_types, and hierarchical graph partitioning for large designs [2103.13060][2512.01463]. True architectural extrapolation in surrogate predictors remains an open challenge; GNN and transformer surrogates outperform simple MLP baselines, but generalization to unseen topologies is actively researched [2511.05615][2408.05314].

Future work is expected to further automate optimal architecture-platform co-design, expand ASIC and embedded software integration, support more complex models (e.g., larger transformers, SSMs), and foster closer integration with quantization/pruning-aware training and neural architecture search workflows [2501.05515][2512.01463].

---

**References:**  
- [2512.01463] "hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardware"  
- [2507.04535] "da4ml: Distributed Arithmetic for Real-time Neural Networks on FPGAs"  
- [2003.06308] "Compressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML"  
- [2311.05716] "ML-based Real-Time Control at the Edge: An Approach Using hls4ml"  
- [2205.07690] "Real-time semantic segmentation on FPGAs for autonomous vehicles with hls4ml"  
- [2207.00559] "Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml"  
- [2402.01047] "Ultra Fast Transformers on FPGAs for Particle Physics Experiments"  
- [2012.01563] "Accelerated Charged Particle Tracking with Graph Neural Networks on FPGAs"  
- [2101.05108] "Fast convolutional neural networks on FPGAs with hls4ml"  
- [2103.05579] "hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices"  
- [2501.14663] "End-to-end workflow for machine learning-based qubit readout with QICK and hls4ml"  
- [2501.05515] "Neural Architecture Codesign for Fast Physics Applications"  
- [2511.05615] "wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation"  
- [2103.13060] "De-specializing an HLS library for Deep Neural Networks: improvements upon hls4ml"  
- [2408.05314] "rule4ml: An Open-Source Tool for Resource Utilization and Latency Estimation for ML Models on FPGA"  
- [2304.06745] "End-to-end codesign of Hessian-aware quantized neural networks for FPGAs and ASICs"  
- [2002.02534] "Fast inference of Boosted Decision Trees in FPGAs for particle physics"  
- [2409.05207] "Low Latency Transformer Inference on FPGAs for Physics Applications with hls4ml"

Source: https://www.emergentmind.com/topics/hls4ml