---
title: 'AXI4MLIR: Host-Accelerator Runtime Optimization'
url: https://www.emergentmind.com/topics/axi4mlir
type: topic
---

# AXI4MLIR: Host-Accelerator Runtime Optimization

Searching arXiv for the cited AXI4MLIR and related MLIR-FPGA papers to ground the article.
AXI4MLIR is an extension of the Multi-Level Intermediate Representation (MLIR) compiler framework for automatic host-driver code generation for custom AXI-based accelerators, with a particular emphasis on linear algebra workloads such as matrix multiplication and convolution. Its central objective is not accelerator synthesis in isolation, but the generation of host-side communication and runtime logic that is aware of accelerator instructions, communication patterns, CPU cache hierarchy, dataflow, and tiling constraints. In the published line of work, AXI4MLIR evolves from a user-driven framework built around accelerator descriptors and an intermediate `accel` dialect [2312.14821], to a system with explicit host-side data-transfer optimizations such as DMA-based allocation, coalescing of DMA transfers, and software pipelining [2402.19184], and then to zero-copy, descriptor-driven data movement that removes redundant heap-to-DMA staging copies [2606.11158].

## 1. Problem domain and design objective

AXI4MLIR was introduced to address a gap left by many accelerator-generation tools: they focus on designing accelerators, but devote comparatively little attention to the host-side driver code required to use those accelerators efficiently [2312.14821]. For custom AXI-based accelerators, the host must allocate and stage memory correctly, pack data into accelerator-specific layouts, send control opcodes and instructions, manage DMA transfers, synchronize send and receive phases, and exploit the CPU memory hierarchy to reduce cache misses and branch overhead. The framework treats these tasks as a compiler problem rather than as handwritten runtime engineering.

The motivating workloads are linear algebra algorithms used in machine learning and scientific computing. In this setting, communication can dominate runtime; different accelerators may require distinct dataflows such as nothing stationary, A-stationary, B-stationary, or C-stationary; and CPU cache-aware tiling and packing can materially affect end-to-end performance [2312.14821]. AXI4MLIR therefore targets the host-accelerator interaction layer, where data movement and loop structure determine whether a custom accelerator is actually beneficial.

The framework is explicitly user-driven. The user specifies accelerator capabilities, supported opcodes, supported opcode flows, tile sizes, data types, and tensor dimensions, while the compiler automates the generation of host code and DMA interactions. This division of labor is significant: AXI4MLIR does not infer arbitrary accelerator internals, but it does convert a structured accelerator description into executable host code and communication logic [2312.14821].

## 2. Compilation flow and intermediate representation structure

The core AXI4MLIR workflow begins with a high-level MLIR program, typically expressed in the `linalg` dialect, and progressively lowers it into host code that drives an AXI DMA runtime [2312.14821]. For matrix multiplication, the paper illustrates a sequence from `linalg.matmul` to `linalg.generic`, then to tiled `scf.for` loops, and finally to accelerator runtime calls. The computational pattern is standard:

$$
C(M,N) = A(M,K) \times B(K,N).
$$

When the accelerator supports a tile such as `MatMul_{4x4x4}`, the loop body of the tiled software version is replaced by accelerator communication.

The end-to-end flow described in the original work is structured as follows. A user writes a high-level MLIR program, provides a configuration file describing CPU cache sizes and types, accelerator capabilities, supported opcodes, supported opcode flows, accelerator dimensions, data types, and tensor dimensions, and then the compiler parses that configuration, identifies compatible `linalg.generic` operations, applies tiling transformations using cache sizes and accelerator dimensions, introduces `accel` dialect operations, lowers those operations to DMA runtime calls, and emits final host code [2312.14821].

Later work makes the lowering sequence more explicit. The 2026 zero-copy study describes the flow as: starting from high-level `linalg` ops such as `linalg.matmul` or `linalg.conv_2d`, converting named ops into `linalg.generic` based on traits such as indexing maps and iterator types, bufferizing tensors into `memref` form, annotating operations with custom accelerator-targeting attributes, lowering into the custom `accel` dialect plus supporting MLIR dialects, applying tiling transformations using `memref` and `scf`, and then lowering `accel` transfers into runtime library calls that control the AXI DMA engine [2606.11158]. In that formulation, AXI4MLIR sits between high-level tensor and linear-algebra representation and the low-level runtime responsible for host-accelerator communication.

The principal architectural point is that AXI4MLIR introduces an intermediate representation for communication. Rather than lowering immediately to LLVM or C, it uses a custom `accel` dialect to represent accelerator initialization, sending of literals and operands, reception of results, and synchronization. The papers argue that this makes transformations easier, dependencies clearer, and communication placement in loop nests more explicit [2312.14821].

## 3. Accelerator description model: attributes, opcode semantics, and the `accel` dialect

AXI4MLIR extends `linalg.generic` with a set of accelerator-oriented attributes and adds a new `accel` dialect that captures host-accelerator communication at a higher abstraction level than raw low-level code [2312.14821]. The added attributes are central to the framework’s configurability.

`dma_init_config` defines parameters needed to initialize the DMA engine for a specific accelerator, with example fields including `id`, `inputAddress`, `inputBufferSize`, `outputAddress`, and `outputBufferSize`. `init_opcodes` defines a flow of opcodes used to initialize or reset the accelerator before execution. `accel_dim` specifies the accelerator’s supported tile size or computation size; for example, `map<(m, n, k) -> (4, 4, 4)>`. `permutation_map` defines loop ordering and thus affects the implied dataflow. `opcode_map` encodes accelerator opcodes as sequences of communication actions. `opcode_flow` defines valid opcode sequences, including nested grouping used as a proxy for communication placement across loop nests [2312.14821].

The syntax presented for `opcode_map` and `opcode_flow` is:

```text
opcode_dict  ::= `opcode_map` `<` opcode_entry (`,` opcode_entry)* `>`
opcode_entry ::= (bare_id | string_literal) `=` opcode_list
opcode_list  ::= `[` opcode_expr (`,` opcode_expr)* `]`
opcode_expr  ::= `send` `(` bare_id `)`
               | `send_literal` `(` integer_literal `)`
               | `send_dim` `(` bare_id `)`
               | `send_idx` `(` bare_id `)`
               | `recv` `(` bare_id `)`

opcode_flow_entry ::= `opcode_flow` `<` flow_expr >
flow_expr ::= `(` flow_expr `)` | bare_id (` ` bare_id)*
```

The communication actions have fixed meanings. `send(arg)` sends data associated with an argument of the `linalg.generic` operation, `send_literal(value)` sends a fixed control literal, `send_dim(arg)` sends a dimension value, `send_idx(arg)` sends an index value, and `recv(arg)` receives a result associated with an argument [2312.14821]. In the matrix multiplication examples, opcodes such as `sA`, `sB`, `cC`, and `rC` correspond to specific sequences of literal and data-transfer actions.

These attributes allow the compiler to express dataflows directly. For matrix multiplication, an example trait set includes `accel_dim = map<(m, n, k) -> (4, 4, 4)>`, `permutation_map = affine_map<(m, n, k) -> (m, k, n)>`, and `opcode_flow = opcode_flow < (sA (sBcCrC)) >`, which encodes an A-stationary flow in which A is sent once at outer scope while B and C communication is repeated in an inner scope [2312.14821]. The paper also lists alternatives such as `((sA sB cC) rC)` for C-stationary behavior and `(sB sA cC rC)` for nothing stationary or a different ordering. A convolution example similarly uses `accel_dim`, `opcode_map`, and `opcode_flow <(sF (sIcO) rO)>` to represent a filter plus output stationary flow.

The `accel` dialect then materializes these communication intents as explicit operations such as `accel.dma_init(...)`, `accel.sendLiteral(...)`, `accel.send(...)`, `accel.sendDim(...)`, and `accel.recv(...)` [2312.14821]. These operations are the substrate on which later host-side optimizations are applied.

## 4. Host memory hierarchy, DMA runtime, and data-transfer optimization

A major contribution of AXI4MLIR is that it does not generate merely correct communication; it generates communication that attempts to exploit the host memory hierarchy. The compiler uses CPU cache sizes from the configuration file, accelerator tile size, and problem dimensions to tile loops so that data is reused in cache and staged efficiently into DMA buffers [2312.14821]. At the runtime level, AXI4MLIR includes a lightweight AXI DMA runtime library that initializes DMA engines, maps memory for input and output staging buffers, copies from `memref` to DMA-accessible buffers, triggers DMA send, waits for completion, and copies received data back into `memref` buffers. The paper reports a size of **55 bytes** for the target ARM SoC [2312.14821].

The framework also specializes `memref` handling. Because MLIR `memref`s carry an allocated pointer, aligned pointer, offset, sizes, and strides, generic MemRef-to-DMA copies can preserve arbitrary layouts. When `strides[N-1] == 1`, however, AXI4MLIR specializes the copy path and uses `std::memcpy(src, dst, size)` instead of recursive element-by-element copying, reducing branch instructions, cache references, and control-flow overhead [2312.14821].

The 2024 extension, presented in "Data Transfer Optimizations for Host-CPU and Accelerators in AXI4MLIR" [2402.19184], studies a flexible stream-based MatMul accelerator with double buffering and identifies two bottlenecks: compute-core utilization below 10% on average, and heap-to-memory-mapped DMA-buffer copy latency as the critical host-side cost. It proposes three host-code data-movement optimizations. First, DMA-based data allocation adds a new attribute or tag, shown as `#dma`, so accelerator-bound `memref`s are allocated directly in DMA-addressable memory. Second, coalescing of DMA transfers merges multiple `accel.send` operations into one variadic send such as `accel.send([%op0,%sA,%sB,%op1])`, reducing fixed DMA launch overhead and synchronization cost. Third, software pipelining rewrites the host loop structure to overlap the accelerator’s load, compute, and store stages and to exploit double buffering [2402.19184].

The 2026 work, "Defeat the Heap: Zero-Copy Data Movement in AXI4MLIR" [2606.11158], pushes this line further by removing the staging copy entirely. It reworks `accel.alloc` so that allocation is modeled using a memory-space-type attribute and an accompanying layout operand, and lowers allocation to `aximlir_dma_alloc()`, which returns `memref` descriptors allocated in the DMA-visible region. The conceptual descriptor shown is:

```cpp
template<typename T, size_t N> struct MemRefDescriptor {
  T *allocated;
  T *aligned;
  intptr_t offset;
  intptr_t sizes[N];
  intptr_t strides[N];
};
```

In the baseline flow, `accel.send` lowers through explicit copies and calls such as `@copy_to_dma_region`, `@dma_start_send`, and `@dma_wait_send_completion`. In the optimized flow, `accel.send` lowers directly from a `memref` or `memref.subview` descriptor to `dma_send_descriptor()` with no intermediate heap-to-DMA copy [2606.11158]. Because `memref.subview` retains the original `allocated` and `aligned` pointers while updating offset, sizes, and strides, the runtime can compute the DMA transfer range directly from descriptor metadata. For non-contiguous tiles, the paper uses scatter-gather DMA, where multiple disjoint source regions are gathered into a contiguous on-device stream and disjoint outputs are scattered back.

## 5. Evaluation results and measured behavior

The original evaluation of AXI4MLIR uses a **PYNQ-Z2 board** with a **Zynq-7000 SoC**, a **dual-core ARM Cortex-A9 CPU** at **650 MHz**, and a library of tile-based accelerators derived from **SECDA-TFLite**, synthesized with AMD/Xilinx Vitis at **200 MHz** [2312.14821]. The accelerators are AXI-Stream connected and micro-ISA or opcode based. The framework is evaluated on matrix multiplication, all convolution layers of **ResNet18**, and an end-to-end **TinyBERT** workload compiled via **Torch-MLIR**.

For matrix multiplication, the paper reports that accelerator offload becomes beneficial when the problem size is at least `dims >= 64`, where `dims = M = N = K`, and when the accelerator tile size is at least `accel_size >= 8` [2312.14821]. Compared with manually optimized driver code, AXI4MLIR-generated drivers achieve **up to 1.65× speedup**, an **average speedup of 1.18×**, **up to 56% reduction in CPU cache references**, and an **average cache-reference reduction of 10%**. For ResNet18 convolution layers, AXI4MLIR is faster on **10 out of 11 layers**, with **1.28× average speedup** and **1.54× max speedup**; one layer slows down by about **10%** because filter height is 1, input channels are small, and the specialized contiguous-copy optimization cannot help sufficiently. For TinyBERT, the reported gains are **3.4× end-to-end speedup**, **18.4× speedup in accelerated MatMul layers**, and the observation that MatMul layers constituted about **75% of the original CPU runtime** [2312.14821].

A concise summary of reported results is:

| Evaluation context | Reported result | Source |
|---|---:|---|
| MatMul drivers vs manual code | up to **1.65×** speedup | [2312.14821] |
| CPU cache references | up to **56%** reduction | [2312.14821] |
| ResNet18 convolution layers | **1.28×** average, **1.54×** max | [2312.14821] |
| TinyBERT end to end | **3.4×** speedup | [2312.14821] |
| Zero-copy main-memory movement | up to **2×** reduction | [2606.11158] |
| Zero-copy end-to-end performance | **1.7×** average improvement | [2606.11158] |

The 2024 MatMul case study adds cycle-level characterization. It reports **average compute-core utilization below 10%**, notes that **Load A takes more cycles than load B**, attributes this to higher startup latency on the first load, and mentions that later transfers can proceed with as little as a **1-cycle delay** once the stream is active [2402.19184]. The conclusion is that the data path, not the MAC array, is the principal throughput limiter in the studied configuration.

The 2026 zero-copy evaluation uses a configurable tiled matrix-matrix multiplication accelerator on a simulated **Versal VP1902 SoC** with **273 GB/s** peak aggregate memory bandwidth over the NoC, **AXI4-based hardened NoC implementation**, **150 ns per-call DMA latency overhead**, and an accelerator engine array capable of **4 TFLOPS** [2606.11158]. It considers non-stationary, A-stationary, and C-stationary dataflows, with \(M = N = K \in \{128, \dots, 1024\}\) and \(T_M = T_N = T_K \in \{16, 32, 64, 128, 1024\}\). The reported outcomes are a reduction in main-memory data movement by up to **2×**, an average **1.7×** improvement in end-to-end performance across all evaluated accelerator sizes and data flows, and **1.7× on average** reductions in both load and store operation latency [2606.11158]. The paper also states that in some configurations with small tiles, transfer cost exceeds compute time by more than **18×**, underscoring that staging-copy overhead can dominate execution.

## 6. Scope, limitations, and relation to adjacent MLIR-based FPGA work

AXI4MLIR has a well-defined scope. It supports SoCs with **ARM CPUs**, accelerators connected via **AXI-Stream**, and systems in which DMA is used for communication [2312.14821]. It does not currently address host-accelerator cache coherence protocols, arbitrary interconnects without adapting the runtime, full scheduling optimization as a separate compiler component, or arbitrary accelerator internals. Most importantly, it does **not** model or optimize the internal accelerator architecture; the accelerator developer must provide supported operations, tile sizes, opcodes, flows, and dimensions [2312.14821]. In this sense, AXI4MLIR is a host-code generation and communication optimization framework, not a complete accelerator synthesis solution.

This boundary is useful when comparing AXI4MLIR to adjacent MLIR-based FPGA research. **Olympus**, introduced for platform-aware FPGA system architecture generation, targets the **system architecture** surrounding accelerators rather than host-driver code. It represents accelerator-system dataflow graphs using an Olympus dialect with `olympus.kernel` nodes and channel constructs such as `olympus.make_channel`, models channel semantics as `stream`, `small`, or `complex`, and performs platform-aware transformations including channel reassignment, replication, bus widening, Iris-based bus optimization, and PLM optimization [2309.12917]. For Xilinx devices, kernels connected to complex channels have an **AXI port** that connects straight to global memory, and the backend uses a **Vivado block diagram** and a Vitis HLS bridge module. This suggests a complementary but distinct layer: Olympus focuses on generated hardware structure and memory-system fit, whereas AXI4MLIR focuses on host-side communication and driver generation.

By contrast, **POM**, "An Optimizing Framework on MLIR for Efficient FPGA-based Accelerator Generation" [2401.05154], is explicitly **not AXI4MLIR**. It is an end-to-end FPGA HLS and compiler framework with a three-layer IR stack, polyhedral semantics via `isl`, a declarative DSL, and automatic design space exploration. The paper states that it does **not** mention AXI4 interfaces, does **not** describe AXI4 memory-mapped communication, and does **not** discuss host-device interaction over AXI4 [2401.05154]. The distinction clarifies a common misconception: not every MLIR-based FPGA compiler is addressing the host-driver and DMA problem that AXI4MLIR takes as its primary target.

Taken together, the AXI4MLIR papers define a coherent research trajectory. The framework begins with user-driven accelerator descriptors, cache-aware tiling, an `accel` dialect, and DMA runtime lowering [2312.14821]; it then identifies concrete host-side utilization bottlenecks and addresses them through DMA-aware allocation, transfer coalescing, and software pipelining [2402.19184]; and finally it extends the dialect and lowering to support zero-copy, descriptor-driven DMA transfers and scatter-gather handling of non-contiguous subviews [2606.11158]. The consistent theme is that efficient use of custom AXI-based accelerators depends not only on arithmetic throughput, but also on compiler-managed communication structure, host memory placement, and DMA semantics.

Source: https://www.emergentmind.com/topics/axi4mlir