Compilation and Execution of an Embeddable YOLO-NAS on the VTA

Abstract: Deploying complex Convolutional Neural Networks (CNNs) on FPGA-based accelerators is a promising way forward for safety-critical domains such as aeronautics. In a previous work, we have explored the Versatile Tensor Accelerator (VTA) and showed its suitability for avionic applications. For that, we developed an initial stand-alone compiler designed with certification in mind. However, this compiler still suffers from some limitations that are overcome in this paper. The contributions consist in extending and fully automating the VTA compilation chain to allow complete CNN compilation and support larger CNNs (which parameters do not fit in the on-chip memory). The effectiveness is demonstrated by the successful compilation and simulated execution of a YOLO-NAS object detection model.

• The paper formalizes an automated framework that converts quantized YOLO-NAS models into VTA-executable binaries using a custom intermediate representation.
• It introduces multiple heuristic matrix partitioning strategies that balance instruction count and memory constraints on FPGA-based accelerators.
• Experimental results demonstrate bit-accurate execution, highlighting potential for DO-178C certification and future optimizations in computation efficiency.

Embedding YOLO-NAS on the VTA: Compilation and Execution Framework

Introduction and Motivation

Deployment of CNNs in safety-critical domains—such as aeronautics—poses formidable challenges due to stringent hardware restrictions and certification requirements. The Versatile Tensor Accelerator (VTA), an open-source FPGA-based platform, offers a compelling hardware solution for accelerated matrix computations, but leveraging its capabilities to run industrial-scale networks like YOLO-NAS requires advanced compilation strategies. This paper formalizes and extends a stand-alone VTA compilation chain, targeting full automation of quantized CNN deployments and focusing on the partitioning, mapping, and execution of large models that exceed accelerator on-chip memory constraints.

Formalization of VTA Matrix Operations

The VTA architecture, comprising concurrent CPU/VTA DRAM access and specialized Load, Compute (GEMM/ALU), and Store modules, operates strictly on quantized matrices and vectors, enforcing a block size $bs$ determined by hardware configuration. To bridge the gap between ONNX-level tensor representations and VTA-executable instructions, the compiler decomposes all matrix operations into fixed-size block matrix operations and formalizes this mapping in an intermediate representation (VTA IR).

This formal layer introduces:

• Explicit semantic mapping of im2row-based transformations from convolutional layers to matrix multiplication.
• One-to-one conversion of block matrix operations into sequences of VTA atomic instructions—GEMM for matrix-matrix, ALU for vector-wise operations.
• Precise indexing schemes connecting elements in original matrices to block representations, ensuring deterministic memory organization and access patterns.

VTA Intermediate Representation and Conversion Logic

The paper rigorously specifies the VTA IR syntax (via EBNF) for describing buffer allocations, matrix loading sequences, block GEMM/ALU operations, and instruction emission. The IR is generated per network layer, supporting both full matrix and partial block decomposition, and enabling automated conversion of ONNX models into VTA-executable binaries. All critical elements of CNN inference—including data reshaping, layer chaining, memory addressing, and partitioning strategy selection—are captured and parameterized in the IR.

Automated Compilation and Layer Chaining

To enable stand-alone execution of extensive CNNs (e.g., YOLO-NAS), the compiler automates the following:

• Parsing ONNX models for topology, parameter extraction, and quantization metadata.
• Backend transformation of tensor layers into block matrices.
• IR generation for each layer, parameterized with partitioning strategies to accommodate on-chip memory overflow.
• Generation of CPU code to orchestrate VTA offloading and inter-layer data formats, including hard-coded memory reshaping routines and optimization of DRAM allocation across all layers.

Matrix Partitioning Strategies

A novel contribution of the paper is the design and implementation of four distinct heuristic matrix partitioning strategies for GEMM operations. These strategies determine how oversized matrices are decomposed into block-sized operations that fit within VTA buffers. Each strategy yields different trade-offs in terms of instruction count, memory access pattern, and potential for parallelism.

(Figure 1)

Figure 1: Decomposing a succession of $bALU{op}$ using partitioning strategies to optimize VTA buffer usage.

Additionally, ALU operations utilize a unique strategy for vector block decomposition, prioritizing minimal buffer overflow and maintaining ordering constraints imposed by inter-operation dependencies.

(Figure 2)

Figure 2: Possible combination between GeMM and ALU matrix partitioning, demonstrating interleaving strategies for optimal offloading.

Experimental Analysis: YOLO-NAS Compilation and Execution

The extended compiler is benchmarked using a quantized YOLO-NAS CNN, featuring 146 operators (108 VTA-compatible). Parsing and IR generation times are modest, and applying Strategy 4 yields over 10M instructions and 9M UOPs for the model. Bit-wise correctness is validated against a Numpy reference implementation, with minor $±1$ differences vs. ONNX Runtime due to differing QLinearConv optimizations.

Key findings include:

• Memory Overhead: Compiled graphs are substantially more compact than ONNX, but biases are expanded from vectors to block matrices to initialize the accumulator buffer, increasing footprint by several orders of magnitude.
• Partitioning Impact: Strategy selection strongly affects instruction count (up to a factor of two), but does not affect total UOP count. Optimal strategy is highly shape-dependent; for some matrix shapes, a suboptimal strategy can result in a 10–14× increase in instruction count.
• Simulation Observations: Pipeline execution time remains tractable, mostly dominated by CPU-driven data manipulation between layers due to floating-point operation requirements.

Implications and Future Directions

The precise formalization and automation enable scalable deployment of industrial-grade CNNs on VTA, with bit-accurate correctness, compact graph representation, and adaptive instruction generation. These results demonstrate VTA's potential for open-source, certifiable accelerator chains in restricted hardware environments.

Practical implications are as follows:

• Certification Feasibility: Stand-alone compilation outputs can facilitate DO-178C certification, bypassing dependencies on frameworks like TVM.
• Dynamic Strategy Optimization: Future enhancements should employ cycle-accurate simulation to optimize partitioning not solely for instruction count, but for latency and throughput.
• Bias Memory Optimization: Mitigating bias expansion via runtime replication or alternative initialization can further improve memory efficiency.
• Full Offloading: Migrating floating-point operations to fixed-point representations would increase operator coverage and reduce CPU dependency.

Theoretically, the formal framework sets the stage for optimal partitioning algorithms, exploring space beyond heuristics and leveraging analytical models for minimal load/store cycles.

Conclusion

The extended VTA compilation chain delivers automated, scalable conversion of quantized ONNX-based CNNs into VTA-executable binaries, with robust correctness and detailed formalization. The implementation of customizable matrix partitioning strategies, efficient IR generation, and comprehensive simulation validates the approach for industrial, certifiable deployment of large-scale models such as YOLO-NAS. Further work on fixed-point representation, bias memory efficiency, and cycle-accurate simulation will continue to advance practical execution and deployment efficiency.