On-FPGA Inference Tools

• On-FPGA inference tools are integrated systems that compile, deploy, and optimize ML models onto FPGA hardware using specialized frameworks and methodologies.
• They utilize diverse architectures—from network-specific streaming accelerators to runtime-flexible overlays—to balance high efficiency with adaptability across varying neural network topologies.
• Key features include memory optimization, explicit control over hardware parameters, and hybrid design flows that combine high-level compilation with detailed hardware-level tuning.

Searching arXiv for recent and foundational papers on FPGA inference tools and deployment flows. {"query":"FPGA inference tools FINN hls4ml OpenCL FPGA CNN inference arXiv", "max_results": 10} On-FPGA inference tools are the software frameworks, compiler flows, hardware-generation methodologies, runtime managers, and deployment utilities used to translate trained machine-learning models into FPGA-resident inference engines or to optimize their feasibility, latency, throughput, and energy. In the literature, the term covers substantially different artifacts: end-to-end mapping frameworks such as fpgaConvNet, open-source flows such as hls4ml and FINN, OpenCL- and HLS-based accelerator-generation methods, deployment-oriented runtimes such as Vitis AI and agent-managed offload systems, and specialized optimization components for memory packing, pre-synthesis estimation, and adaptive execution (Venieris et al., 2017, Borras et al., 2022, Dua et al., 2020, Rahimifar et al., 2024). A persistent theme is that FPGA inference tooling is shaped less by abstract model execution alone than by streamability, on-chip memory organization, quantization, and the boundary between static hardware specialization and runtime flexibility.

1. Tool classes and architectural scope

The literature does not treat on-FPGA inference tools as a single compiler category. One major class consists of network-specific streaming accelerators, where the hardware graph follows the neural-network graph and layers are implemented as dedicated logic or tightly specialized HLS blocks. fpgaConvNet models CNN deployment with Synchronous Dataflow (SDF), explores graph partitioning, coarse-grained folding, fine-grained folding, and weights reloading, and targets throughput, latency, or multiobjective criteria under FPGA resource constraints (Venieris et al., 2017). FINN and hls4ml, as used in the MLPerf Tiny study, generate configurable, spatial dataflow architectures tailored for speed and efficiency and support quantization-aware co-design from training to board-level deployment (Borras et al., 2022).

A second class is runtime-flexible accelerator frameworks whose compiled hardware is intended to support multiple models or profiles without regenerating the FPGA image. Systolic-CNN is presented as an OpenCL-defined scalable, run-time-flexible FPGA accelerator architecture whose hardware structure is invariant to the CNN model and can be time-shared across different CNNs at runtime without recompiling the FPGA kernel hardware nor reprogramming the FPGA (Dua et al., 2020). The ONNX-to-hardware adaptive flow extends this idea toward approximate computing by generating multi-profile engines from QONNX models and merging profiles through MDC so that runtime profile switching trades power against accuracy (Manca et al., 2024).

A third class is software-oriented HLS flows. The LegUp-based VGG-16 system is synthesized from a multi-threaded C program using Pthreads, where each software thread becomes a concurrent hardware block and FIFO communication becomes hardware FIFOs; the accelerator reaches 138 effective GOPS on a mid-sized Intel Arria 10 SoC FPGA (Kim et al., 2018). FFCNN and Systolic-CNN represent Intel OpenCL-centered design methodologies in which kernel structure, channels/pipes, and loop organization are written with FPGA compilation behavior in mind rather than treated as GPU-style kernels (Keddous et al., 2022, Dua et al., 2020).

A fourth class is deployment frameworks with strong host-side orchestration. FusionAccel combines an RTL CNN engine, host-controlled execution, Python preprocessing scripts, and layer-by-layer runtime programmability, but it remains closer to a configurable hardware-software stack than to a full automatic compiler (Shi, 2019). AI-FPGA Agent goes further toward runtime-managed heterogeneous execution by introducing a software agent that dynamically partitions workloads between CPU and FPGA, schedules offload, and manages data transfers through a high-level API (Yunusoglu et al., 27 Jan 2026).

A plausible implication is that “on-FPGA inference tool” is best understood as an umbrella category spanning compilers, generator libraries, runtimes, and optimization passes rather than a single software abstraction layer.

2. Execution models: custom dataflow, overlays, and streaming I/O

The most consequential design split is between topology-specialized custom dataflow and runtime-flexible or overlay-like engines. FINN-style custom dataflow keeps weights on chip, specializes arithmetic precision per layer, and maps layer or layer fragments to dedicated logic. In “Memory-Efficient Dataflow Inference for Deep CNNs on FPGA,” this style is explicitly contrasted with overlay accelerators, which are more general but require repeated movement of activations and weights between compute and off-chip memory (Petrica et al., 2020). The attraction of the custom dataflow class is low latency and high energy efficiency when weights fit on chip; its weakness is reduced topology flexibility and greater sensitivity to BRAM/URAM packing, floorplanning, and timing closure.

Streaming is the dominant execution model across many toolflows. fpgaConvNet represents CNN execution as a streaming SDF graph (Venieris et al., 2017). SNL is designed with a streaming data approach, optimizing data flow between layers while minimizing buffering requirements, and it supports full redeployment of weights and biases without requiring resynthesis after training (Herbst et al., 2023). In the MLPerf Tiny implementations, both hls4ml and FINN produce streaming layer pipelines, though hls4ml generates a single HLS top-level accelerator while FINN generates per-layer HLS/IP blocks that are stitched into a full streaming design (Borras et al., 2022).

Streaming also appears outside CNNs. The PCIe-streamed XGBoost deployment on Alveo U280 uses XDMA in streaming mode so that inputs are delivered directly to application logic and outputs are sent back without intermediate storage in FPGA memory, enabling a record-level pipeline rather than a copy-process-copy batch engine (Manavar et al., 2021). A different form of stream-centric design appears in the RISC-V ISA-extension system, where FPGA.VCONV, FPGA.GEMM, and FPGA.RELU expose operator kernels as custom instructions, making FPGA-resident accelerators software-visible at the instruction-set level rather than only through coarse memory-mapped calls (Parameshwara et al., 10 Nov 2025).

A common misconception is that runtime flexibility is simply a stronger version of specialization. The literature instead shows a trade: Systolic-CNN preserves model invariance and achieves up to 100% DSP utilization on Arria 10 while accepting the costs of FP32 and an architecture constrained to supported CNN operators (Dua et al., 2020). By contrast, fully specialized dataflow systems often achieve lower latency and better on-chip reuse, but only after fixing topology, folding, and quantization.

3. Front ends, intermediate representations, and hardware generation

The front-end diversity of on-FPGA inference tools is unusually broad. fpgaConvNet uses Caffe as the front end and converts the network into a pair of SDF models, one for workload and one for hardware mapping (Venieris et al., 2017). hls4ml and FINN are increasingly connected through QONNX, which is used as a quantization-aware interchange format between training libraries such as QKeras and Brevitas and hardware compilers (Borras et al., 2022). The ONNX-to-hardware adaptive flow also starts from QONNX, then uses an extended ONNXParser, an HLS Writer, Vitis HLS, and MDC to generate either a single streaming engine or a merged multi-profile adaptive engine (Manca et al., 2024).

Other toolchains expose different source abstractions. SNL uses a Keras-like API implemented as header-only C++ templates within Xilinx Vitis HLS, so the model structure is specified directly in C++ while weights are produced externally, for example in Keras, and then loaded at runtime (Herbst et al., 2023). The LegUp-based system uses a graph of software threads and FIFOs as its effective IR, with LEGUP_PTHREAD_FIFO and pthread_fifo_read/pthread_fifo_write directly mapping to communicating hardware blocks (Kim et al., 2018). MAFIA begins from SEEDOT DSL or a subset of TensorFlow translated to SEEDOT, builds a matrix DFG, and generates Verilog directly rather than relying on C-HLS as the final implementation vehicle (Ghanathe et al., 2021).

At the deployment end, toolchains differ sharply in how much automation they expose. Vitis AI compiles supported PyTorch-derived models to .xmodel for execution on DPUCZDX8G B4096 and is relatively structured, but it requires DPU-supported operators and INT8 inference (Antunes et al., 14 Mar 2026). The Vitis HLS flow in the same study converts PyTorch to ONNX, ONNX to C through ONNX2C, then synthesizes custom IP, which expands operator coverage to sigmoid and 3D layers but requires much more manual integration (Antunes et al., 14 Mar 2026).

This suggests that front-end choice is not merely syntactic. It determines which quantization metadata survive to hardware generation, which operators are legally expressible, and whether deployment centers on a model-specific datapath, a runtime-programmable kernel set, or a fixed microarchitecture such as a DPU.

4. Memory layout, feasibility estimation, and utilization-aware optimization

Memory organization is frequently the binding constraint in FPGA inference deployment. In FINN-style dataflow, throughput-driven parameters such as $N_{PE}$ and $N_{SIMD}$ create wide, shallow logical memories that pack poorly into BRAM, which is why “Memory-Efficient Dataflow Inference for Deep CNNs on FPGA” introduces Frequency Compensated Memory Packing (FCMP). FCMP combines memory overclocking with logical buffer packing and shows up to 30% reduction in OCM utilization without loss of inference throughput, including board portability from Xilinx Zynq 7020 to 7012S and from Alveo U250 to the smaller Alveo U280 with less throughput loss than folding (Petrica et al., 2020).

A related optimization stage is Evolutionary Bin Packing for Memory-Efficient Dataflow Inference Acceleration on FPGA, which treats parameter-memory placement as a constrained bin-packing problem and uses hybrid GA/SA methods with Next-Fit Dynamic (NFD). Its hybrid mappers converge to optimal solutions in a matter of seconds for all CNN use-cases, achieve an increase of up to 65% in OCM utilization efficiency for deep CNNs, and are up to 200× faster than prior simulated annealing approaches (Kroes et al., 2020). This work is not a complete compiler, but it is a critical toolflow component between architectural folding decisions and actual RTL/HLS realization.

Pre-synthesis feasibility estimation has become a distinct tooling category. rule4ml is an open-source estimator for hls4ml-compatible networks that predicts BRAM, DSP, FF, LUT, and inference latency in clock cycles before synthesis. It reports validation $R^2$ scores ranging between 0.8 and 0.98 and sMAPE values between 10% and 30%, providing a fast screen for whether a design is likely to fit on Pynq-Z2, ZCU102, or Alveo-U200 (Rahimifar et al., 2024).

Utilization-aware optimization also extends beyond memory packing. Continuous-Flow Data-Rate-Aware CNN Inference on FPGA observes that stride-$>1$ convolutions and pooling reduce output data rate, leaving naively unrolled hardware idle. Its interleaving and sharing method aims for hardware utilization close to 100% and shows that arithmetic logic can be reduced enough to implement complex CNNs like MobileNet on a single FPGA with high throughput (Habermann et al., 16 Jan 2026).

A plausible implication is that, in mature FPGA inference flows, model compilation and memory-layout optimization are inseparable stages.

5. Runtime flexibility, heterogeneous orchestration, and deployment regimes

Runtime flexibility in FPGA inference does not have a single implementation pattern. Systolic-CNN achieves it through a CNN-model invariant 1-D systolic array whose runtime parameters include filter size, stride, and padding, while compile-time parameters $pe\_num$, $vec\_fac$, and $reuse\_fac$ adapt the hardware to a target FPGA (Dua et al., 2020). The adaptive ONNX-to-hardware flow instead generates multiple approximate profiles, then uses MDC to merge them into a single engine so that runtime switching can trade 5% power saving against 1.5% accuracy drop in the demonstrated adaptive engine (Manca et al., 2024).

A more heterogeneous variant is AI-FPGA Agent, where a runtime software agent partitions a model layer-by-layer or subgraph-by-subgraph between CPU and FPGA, schedules compute-intensive layers for hardware offload, and manages data transfers. Its reported CNN benchmark gives 3.5 ms/image, 284.7 images/s, 28.0 W, and 10.17 images/s/W for the FPGA system, compared with 40.2 ms/image and 24.8 images/s on CPU (Yunusoglu et al., 27 Jan 2026). The same paper, however, contains an inconsistency between the narrative claim of “2–3× higher energy efficiency than GPU implementations” and the printed table, which supports a larger ratio; this is an example of why deployment frameworks should be read together with their detailed tables rather than only their abstracts (Yunusoglu et al., 27 Jan 2026).

The space-use-case study on AMD ZCU104 makes the complementarity of deployment regimes especially explicit. Vitis AI with DPUCZDX8G B4096 is superior when the model fits the DPU’s operator set and INT8 quantization model, reaching up to 34.16× higher inference rate than the embedded ARM CPU baseline. Vitis HLS is the fallback when the network contains unsupported operators such as sigmoid, comparators, and 3D convolution/pooling, but the paper’s “naive” HLS mappings are often much slower on larger models because of limited parallelization and off-chip-memory dependence (Antunes et al., 14 Mar 2026).

The same flexibility-versus-specialization pattern now appears in post-CNN systems. TerEffic introduces a fully on-chip multi-card design and an HBM-assisted single-card design for ternary LLM inference, while SpecMamba couples speculative decoding, hybrid backtracking, FIFO-based tree verification, and a linear-parallel/SSM-sequential dataflow for Mamba inference (Yin et al., 23 Feb 2025, Zhong et al., 24 Sep 2025). These systems suggest that runtime-managed or memory-regime-aware deployment is becoming a general FPGA inference theme, not one limited to CNNs.

6. Limits, misconceptions, and current directions

The literature repeatedly shows that FPGA inference tools are rarely “push-button.” The LegUp study notes that achieving loop pipelines with II = 1 required removing control flow, replacing conditionals with ternary operators, and splitting large controllers to avoid huge FSMs (Kim et al., 2018). FFCNN likewise argues that OpenCL code must be written in an FPGA-native style—single-work-item compute kernels, channel-connected stages, and loop flattening—because merely porting CPU/GPU OpenCL code to FPGA does not produce efficient inference (Keddous et al., 2022).

Another misconception is that all successful FPGA inference tools are DNN compilers. MAFIA is a direct-to-Verilog compiler for compact IoT models such as BONSAI and PROTONN, and it reports that MAFIA-generated programs outperform the best-performing variant of a commercial HLS compiler by 2.5× on average on a Xilinx Arty board (Ghanathe et al., 2021). The PCIe-streamed boosted-tree deployment on Alveo U280 reaches 65.8 M inf/s and about 25× better energy efficiency than CPU for a shallow recommendation model, but the paper is careful to state that this benefit depends on II = 1, compact 4-bit feature encoding, and host software capable of sustaining the stream (Manavar et al., 2021).

The most recent large-model papers extend on-FPGA inference tooling into sequence models and LLMs. TerEffic reports 16,300 tokens/second and 455 tokens/second/W for a 370 M-parameter fully on-chip ternary model, and 521 tokens/s with 16 tokens/s/W for a 2.7B-parameter HBM-assisted model (Yin et al., 23 Feb 2025). SpecMamba reports 2.27× speedup over GPU baselines and 2.85× improvement compared to prior FPGA solutions for Mamba with speculative decoding (Zhong et al., 24 Sep 2025). These numbers indicate a clear shift: FPGA inference tools are no longer confined to small CNNs, but their success still depends on aggressive co-design among quantization, memory placement, execution schedule, and runtime control.

The field therefore remains characterized by a technical trade. Greater automation generally increases usability, but the highest efficiencies still come from flows that expose hardware structure—stream widths, folding, BRAM/URAM organization, custom arithmetic, DMA schedules, or even ISA extensions—to the deployment engineer. A plausible implication is that future on-FPGA inference tools will converge not by eliminating these controls, but by making them explicit, composable, and analyzable across compiler, runtime, and architecture layers.