---
title: ARM Compute Library for Embedded Inference
url: https://www.emergentmind.com/topics/arm-compute-library-arm-cl
type: topic
---

# ARM Compute Library for Embedded Inference

Searching arXiv for relevant ARM Compute Library papers to ground the article.
ARM Compute Library (ACL, also referred to as ARM-CL) is a collection of high-performance kernels for ARM Cortex-A CPUs and ARM Mali GPUs that has been used as a substrate for embedded neural-network inference, initially with an emphasis on CNN workloads and, in later work, with native support for transformer execution on ARM-based heterogeneous multi-processor system-on-chips (HMPSoCs). In the published studies considered here, ACL is characterized by hand-tuned NEON and OpenCL kernels, low framework overhead, and the ability to assemble inference-only runtimes with explicit control over operator placement, memory movement, and graph structure [1704.03751], [2606.02836].

## 1. Historical scope and operator coverage

The 2017 case study presents ACL as a set of building blocks for implementing an embedded CNN inference engine from scratch. Its stated operator coverage at that time comprised Activation, Convolution, Fully Connected, Locally Connected, Normalization, Pooling, and Softmax. The study used the CPU/NEON path exclusively; GPU/OpenCL was mentioned as part of ACL’s general scope but was not evaluated there [1704.03751].

The 2026 transformer study describes a more specific limitation in the then-current baseline, ARM-CL v21.02: despite strong coverage of convolution, GEMM, activation, pooling, and softmax, it lacked native transformer operators such as multi-head attention (MHA), scaled dot-product attention (SDPA), layer normalization, positional embedding, and fused matmul+bias+activation patterns pervasive in transformer blocks. Prior to that extension, transformer users had to assemble these behaviors with generic kernels at significant overhead or fall back to frameworks that underperform on ARM-based HMPSoCs [2606.02836].

Taken together, these studies define ACL less as a monolithic end-user framework than as a performance-oriented operator library. A plausible implication is that ACL’s practical value depends strongly on whether the target model family is well covered by its kernel set: CNNs were already a good fit in the earlier period, whereas transformer deployment required substantive kernel additions before comparable efficiency was attainable.

## 2. ACL as a substrate for custom embedded inference engines

The 2017 case study used ACL to build a SqueezeNet inference engine on a low-cost ARMv7 SoC with four ARM v7 cores at 1 GHz, 512 MB RAM, and peak power of approximately 3 W. The authors report that porting TensorFlow required “days” of dependency porting and about a week to obtain a working build on the bare-metal SoC, whereas building with ACL required “much less development time” for this simple model class [1704.03751].

The implementation strategy was explicitly inference-oriented. The SqueezeNet fire module was assembled from ACL’s core operators, and the implementation “eliminates the need for extra memory copy otherwise needed for [the] concatenation operation.” At that time ACL lacked dropout and global pooling, so dropout was removed during inference and compensated by an attenuation coefficient after pool10, while global pooling was implemented separately. The paper emphasized NEON-enabled ACL kernels for all core operators, avoidance of unnecessary memory copies across layers, and a minimal graph stripped of training-time layers when safe [1704.03751].

On that platform, the ACL-based SqueezeNet engine reported lower single-image latency than TensorFlow under matched conditions. The measured latencies were $T_{TF}=420$ ms and $T_{ACL}=320$ ms, giving a speedup
$$
S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times
$$
and a percentage latency reduction
$$
\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.
$$
The paper described this as “25%,” which approximates the computed 23.81%. Throughput correspondingly increased from approximately $2.38$ img/s to $3.13$ img/s. CPU utilization was reported as approximately $75\%$ for TensorFlow versus approximately $90\%$ for ACL, while runtime memory usage remained modest at approximately $9$ MB for TensorFlow and approximately $10$ MB for ACL [1704.03751].

This result is often interpreted as evidence that embedded inference always favors custom runtimes. The paper is narrower than that: it argues that for simple embedded CNN workloads, especially on constrained or bare-metal systems, a lean ACL-based engine may outperform and out-develop a ported general-purpose framework, but it also cautions that as model complexity grows, porting a mature framework may become more efficient [1704.03751].

## 3. Native transformer support and mathematical structure

The 2026 extension broadens ACL from a CNN-centered kernel library to a native transformer inference substrate. The implemented operator set includes token and positional embeddings; linear projections for $Q/K/V$; SDPA with scaling, masking, softmax, and $AV$ matmul; multi-head attention composition; layer normalization with learnable $\gamma/\beta$ and $\epsilon$; feed-forward MLPs with two linear layers and GELU; residual add; and row-wise softmax. Inference-only dropout is elided [2606.02836].

The principal transformer equations implemented in that work are:
$$
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V
$$
$$
\mathrm{MHA}(X)=\mathrm{Concat}(h_1,\dots,h_H)W_O,\qquad
h_i=\mathrm{Attention}(XW_Q^{(i)},XW_K^{(i)},XW_V^{(i)})
$$
$$
y=\left(\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}\right)\cdot \gamma+\beta,\qquad
\mu=\frac{1}{D}\sum_j x_j,\qquad
\sigma^2=\frac{1}{D}\sum_j (x_j-\mu)^2
$$
and
$$
\mathrm{GELU}(x)=0.5x\left(1+\tanh\left(\sqrt{\frac{2}{\pi}}\left(x+0.044715x^3\right)\right)\right).
$$

The same study provides a compact complexity model. With sequence length $L$, hidden size $D$, heads $H$, and head dimension $d_k=D/H$, the attention block has rule-of-thumb FLOP count
$$
8LD^2+4L^2D,
$$
while the FFN contributes
$$
4LDD_{ff}.
$$
For BERT-base, with $D=768$ and $D_{ff}=3072$, the per-layer FFN dominates attention when $L$ is small. This complexity decomposition is central to the later scheduling rules: GEMM-heavy FFN and projection stages are compute-intensive, whereas reshape, transpose, masking, softmax, and normalization phases are comparatively memory-bound [2606.02836].

A significant conceptual shift follows from this extension. Earlier ACL usage relied on composing CNN-style primitives into inference graphs. The transformer work instead adds domain-specific kernels that encode the structure of attention blocks directly, reducing the need to emulate transformer behavior through generic operators.

## 4. Kernel organization, data layout, and precision modes

The transformer extension introduces both NE (CPU) and CL (GPU) kernels. Among the named components are NE/CL embedding lookup kernels; NEGEMM and CLGEMM for projections and FFN; fused NEGEMMBiasGELU and CLGEMMBiasGELU; BatchedQKMatMulScaleMask; NESoftmax and CLSoftmax; BatchedAVMatMul; ConcatHeads; NELayerNormalization and CLLayerNormalization; NEResidualAdd and CLResidualAdd; and NEGELU and CLGELU [2606.02836].

Sequences are represented as 2D tensors with shape $(L,D)$ in row-major form, mapped to ACL tensor dimensions as width $=D$, height $=L$, channels $=1$. For multi-head operations, the data are reshaped to $(H,L,d_k)$ and permuted between $(L,H,d_k)$ and $(H,L,d_k)$ to maximize contiguous access along $d_k$. On the CPU side, NEON blocking interleaves panels in GEMM and uses FMA-heavy inner loops with register blocking such as $4\times 4$ or $8\times 2$, depending on $D$, cache pressure, and $L$. On Mali GPUs, CLGEMM uses tiled matmul with $K$-blocking; for $D$ in $[384,768]$, a typical choice is $M=64$, $N=64$, $K=16$, adapted based on $L$ to keep tiles in local memory below $32$ KB. Padding strides to multiples of $16$ floats is used to avoid bank conflicts [2606.02836].

Precision support is heterogeneous. FP32 is the default and is stated to ensure numerical equivalence to reference implementations. FP16/BF16 are supported where ACL offers half or bfloat types; FP16 reduces memory footprint and improves GPU throughput, while BF16 is recommended on CPU NEON where available to preserve dynamic range. INT8 (QASYMM8) and INT16 (QSYMM16) are supported through ACL quantized GEMM paths for FFN and linear layers, with per-tensor and per-channel schemes depending on the layer and offline calibration by KL-divergence or min/max. The recommended transformer deployment mode on edge devices is mixed precision: INT8 for linear/FFN and FP16/FP32 for attention softmax and layer normalization [2606.02836].

Memory management is treated as a first-class optimization target. The implementation pre-allocates tensors for all layer outputs, reuses buffers when topologically safe, and exploits shared host memory between CPU and GPU through OpenCL shared buffers. At CPU↔GPU boundaries, “shared” tensors are allocated in an OpenCL buffer backed by host-visible memory, enabling zero-copy hand-off and eliminating explicit memcpy between graph partitions. Consecutive same-device layers use single-target tensors to minimize runtime checks and mapping overhead [2606.02836].

## 5. Cooperative CPU–GPU execution on ARM-based HMPSoCs

The reference platform for the transformer study is the Khadas VIM3 BASIC with an Amlogic A331D SoC fabricated at 12 nm, combining a big.LITTLE CPU and a Mali G52 GPU. The high-performance runs used the quad Cortex-A73 big cluster at 2.2 GHz; the dual Cortex-A53 little cluster at 1.8 GHz was present but not used. The ISA was ARMv8-A with 128-bit NEON SIMD. The GPU was a quad-core Mali G52 of the Bifrost family at approximately 0.8 GHz. The software stack comprised Ubuntu 22.04.4, Linux kernel v4.9, ARM-CL v21.02 extended in the authors’ fork, and OpenCL 3.0. An empirical platform characteristic highlighted in the study is the asymmetry between CPU and GPU cache sizes: CPU L2 cache of 2 MB versus GPU L2 cache of approximately 128 KB [2606.02836].

The scheduler is formulated around arithmetic intensity,
$$
AI=\frac{\mathrm{FLOPs}}{\mathrm{Bytes\ read+writes}},
$$
with memory-intensive operators defined as $AI<4$ FLOPs/byte and compute-intensive operators as $AI\ge 4$ FLOPs/byte. On the reported platform, the decision rule is: if $L\ge 256$, assign all layers to CPU; otherwise map Embedding, SDPA softmax + masking + reshape + concat, and Add+LayerNorm to CPU, while mapping $Q/K/V$ projections, FFN matmuls, and the output projection $W_O$ to GPU. The stated rationale is that memory-intensive operations benefit from CPU cache behavior and low-overhead scalar/vector control, while highly parallelizable GEMMs benefit from the GPU [2606.02836].

The execution pipeline is explicitly cooperative. CPU prepares embeddings and SDPA reshape while GPU performs $Q/K/V$ GEMMs; shared buffers carry intermediate tensors without memcpy; the host avoids blocking except at hand-off points; OpenCL events establish dependencies; and CPU-side NE kernels run on a thread pool. The implementation assembles blocks programmatically rather than through ACL’s Graph API in order to retain fine-grain control over placement and zero-copy buffers. The authors state that if the Graph API is used, explicit subgraph partitioning and replacement of Sender/Receiver with shared CLTensor buffers from their fork are recommended [2606.02836].

Thermal and DVFS effects are treated as operational constraints rather than afterthoughts. The study reports variability from thermal throttling and frequency scaling on embedded SoCs, and therefore recommends pinning big cores for NE workloads, avoiding GPU saturation when $L$ is large, keeping GPU kernels short and tiled, and using cooperative scheduling to reduce peak power and improve sustained throughput [2606.02836].

## 6. Empirical performance, numerical behavior, and robustness

The transformer evaluation covered BERT-base, DistilBERT, MobileBERT, SqueezeBERT, and GPT-2 small. Unless noted otherwise, reported results use $L=32$, batch size $1$, default FP32, and causal masks for GPT-2. Baselines were TVM v0.18 on CPU and GPU and ExecuTorch on CPU; ExecuTorch lacked Mali GPU support. The comparison controlled for equivalent models, matched weight formats, and disabled training-only operations such as dropout [2606.02836].

The headline results are substantial. For single-processor inference, the extended ARM-CL achieved a $2.34\times$ average speedup versus TVM on CPU across models and a $2.23\times$ average speedup versus TVM on GPU. Cooperative CPU-GPU execution reduced latency by up to $15.72\%$ relative to the best single-processor ARM-CL execution and was reported to win across all tested transformers. Representative end-to-end latencies, in milliseconds at $L=32$ and batch $=1$, were: BERT-base $1789.85$ on CPU, $835.62$ on GPU, and $757.66$ cooperatively; DistilBERT $899.57$, $412.11$, and $379.62$; MobileBERT $400.92$, $478.08$, and $357.42$; SqueezeBERT $129.38$, $163.97$, and $109.06$; and GPT-2 small $2417.39$, $570.44$, and $507.86$ [2606.02836].

Layer-level profiling for BERT-base at $L=32$ clarifies the source of these gains. On CPU, Embedding was $0.45$ ms, Attention Linear $25.31$ ms, SDPA $3.13$ ms, FF $121.50$ ms, and Add+Norm $0.32$ ms. On GPU, Embedding rose to $45.23$ ms, Attention Linear dropped to $12.41$ ms, SDPA was $4.05$ ms, FF fell to $45.59$ ms, and Add+Norm was $0.91$ ms. The study therefore attributes the improvement to mapping FF and Attention Linear to GPU while keeping memory-bound Embedding, SDPA reshape/softmax, and Add+Norm on CPU [2606.02836].

The numerical validation is correspondingly explicit. Outputs were checked against reference implementations in PyTorch and TVM; FP32 paths matched within machine precision, while FP16 showed the small deviations expected for reduced precision. Stability measures included log-sum-exp stabilization for softmax row reductions, attention scaling by $1/\sqrt{d_k}$, and layer norm epsilon values such as $\epsilon=10^{-5}$. Causal masks for GPT-2 were implemented by setting upper-triangular positions to $-\infty$ before softmax, and padding masks were handled in BatchedQKMatMulScaleMask. Edge cases addressed in the paper include long sequences up to $L=512$, variable sequence lengths through max-$L$ pre-allocation and slicing, and support for both causal and bidirectional attention [2606.02836].

## 7. Limitations, portability, and research trajectory

The principal limitations reported for ACL-based transformer inference are memory capacity, GPU cache size, and quantization complexity. On typical embedded DRAM capacities of $2$–$4$ GB, BERT-base with approximately $300$ MB of weights fit on the evaluated board, whereas BERT-large with approximately $1$ GB of weights exceeded practical limits and reportedly crashed on the VIM3 setup. The Mali G52’s approximately $128$ KB L2 cache limited gains when working sets exceeded local cache, and the study therefore recommends cooperative or CPU-only execution for $L\ge 256$ [2606.02836].

Quantization is presented as useful but nontrivial. The transformer paper supports INT8 and INT16 GEMM paths and recommends mixed precision, explicitly retaining attention softmax and layer normalization in FP16/FP32 for numerical stability. This is consistent with the earlier CNN case study, which showed that partial quantization inside TensorFlow made convolution kernels approximately $25\%$ faster but increased end-to-end inference time by more than $100$ ms because re-quantize/de-quantize overhead dominated the gain. The shared lesson is that quantization must be designed end to end rather than inserted piecemeal [1704.03751], [2606.02836].

Portability is described with caution. The 2026 scheduling heuristics target Cortex-A73 plus Mali G52 and are said to generalize to similar big cores and Bifrost GPUs with shared host memory, but boards with larger GPU L2 caches may preserve GPU-only advantages at larger sequence lengths. The proposed future directions are kernel fusion beyond GEMM+Bias+GELU, FlashAttention-style implementations for reduced memory traffic at long $L$, SVE2 exploitation on newer cores, Vulkan/Compute frontends where OpenCL is limited, and graph-level scheduling and autotuning integrated with the ACL Graph API. The transformer extensions and examples were released as open source under the MIT license at `https://github.com/dondavan/fast-transformer-on-acl` [2606.02836].

Within the available literature, ARM Compute Library therefore emerges as a kernel-centric inference substrate whose effectiveness depends on close alignment between model structure, operator coverage, and platform-aware scheduling. Its early documented strength lay in lean CNN runtimes on constrained ARM CPUs; its later development extends that role to transformer inference on ARM-based HMPSoCs through native attention kernels, zero-copy CPU–GPU cooperation, and explicit management of arithmetic intensity, cache behavior, and runtime overhead [1704.03751], [2606.02836].

Source: https://www.emergentmind.com/topics/arm-compute-library-arm-cl