ARM Compute Library for Embedded Inference

• ARM Compute Library (ARM-CL) is a collection of optimized kernels for ARM Cortex-A CPUs and Mali GPUs, designed to enable efficient embedded neural network inference.
• It leverages hand-tuned NEON and OpenCL implementations to minimize framework overhead, offering explicit control over operator placement, memory movement, and graph structure.
• Recent extensions add native transformer support and cooperative CPU–GPU execution strategies, achieving significant latency improvements and throughput gains on HMPSoCs.

Searching arXiv for relevant ARM Compute Library papers to ground the article. ARM Compute Library (ACL, also referred to as ARM-CL) is a collection of high-performance kernels for ARM Cortex-A CPUs and ARM Mali GPUs that has been used as a substrate for embedded neural-network inference, initially with an emphasis on CNN workloads and, in later work, with native support for transformer execution on ARM-based heterogeneous multi-processor system-on-chips (HMPSoCs). In the published studies considered here, ACL is characterized by hand-tuned NEON and OpenCL kernels, low framework overhead, and the ability to assemble inference-only runtimes with explicit control over operator placement, memory movement, and graph structure (Sun et al., 2017, Xu et al., 1 Jun 2026).

1. Historical scope and operator coverage

The 2017 case study presents ACL as a set of building blocks for implementing an embedded CNN inference engine from scratch. Its stated operator coverage at that time comprised Activation, Convolution, Fully Connected, Locally Connected, Normalization, Pooling, and Softmax. The study used the CPU/NEON path exclusively; GPU/OpenCL was mentioned as part of ACL’s general scope but was not evaluated there (Sun et al., 2017).

The 2026 transformer study describes a more specific limitation in the then-current baseline, ARM-CL v21.02: despite strong coverage of convolution, GEMM, activation, pooling, and softmax, it lacked native transformer operators such as multi-head attention (MHA), scaled dot-product attention (SDPA), layer normalization, positional embedding, and fused matmul+bias+activation patterns pervasive in transformer blocks. Prior to that extension, transformer users had to assemble these behaviors with generic kernels at significant overhead or fall back to frameworks that underperform on ARM-based HMPSoCs (Xu et al., 1 Jun 2026).

Taken together, these studies define ACL less as a monolithic end-user framework than as a performance-oriented operator library. A plausible implication is that ACL’s practical value depends strongly on whether the target model family is well covered by its kernel set: CNNs were already a good fit in the earlier period, whereas transformer deployment required substantive kernel additions before comparable efficiency was attainable.

2. ACL as a substrate for custom embedded inference engines

The 2017 case study used ACL to build a SqueezeNet inference engine on a low-cost ARMv7 SoC with four ARM v7 cores at 1 GHz, 512 MB RAM, and peak power of approximately 3 W. The authors report that porting TensorFlow required “days” of dependency porting and about a week to obtain a working build on the bare-metal SoC, whereas building with ACL required “much less development time” for this simple model class (Sun et al., 2017).

The implementation strategy was explicitly inference-oriented. The SqueezeNet fire module was assembled from ACL’s core operators, and the implementation “eliminates the need for extra memory copy otherwise needed for [the] concatenation operation.” At that time ACL lacked dropout and global pooling, so dropout was removed during inference and compensated by an attenuation coefficient after pool10, while global pooling was implemented separately. The paper emphasized NEON-enabled ACL kernels for all core operators, avoidance of unnecessary memory copies across layers, and a minimal graph stripped of training-time layers when safe (Sun et al., 2017).

On that platform, the ACL-based SqueezeNet engine reported lower single-image latency than TensorFlow under matched conditions. The measured latencies were $T_{TF}=420$ ms and $T_{ACL}=320$ ms, giving a speedup

$$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$

and a percentage latency reduction

$$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$

The paper described this as “25%,” which approximates the computed 23.81%. Throughput correspondingly increased from approximately $2.38$ img/s to $3.13$ img/s. CPU utilization was reported as approximately $75\%$ for TensorFlow versus approximately $90\%$ for ACL, while runtime memory usage remained modest at approximately $9$ MB for TensorFlow and approximately $10$ MB for ACL (Sun et al., 2017).

This result is often interpreted as evidence that embedded inference always favors custom runtimes. The paper is narrower than that: it argues that for simple embedded CNN workloads, especially on constrained or bare-metal systems, a lean ACL-based engine may outperform and out-develop a ported general-purpose framework, but it also cautions that as model complexity grows, porting a mature framework may become more efficient (Sun et al., 2017).

3. Native transformer support and mathematical structure

The 2026 extension broadens ACL from a CNN-centered kernel library to a native transformer inference substrate. The implemented operator set includes token and positional embeddings; linear projections for $T_{ACL}=320$0; SDPA with scaling, masking, softmax, and $T_{ACL}=320$1 matmul; multi-head attention composition; layer normalization with learnable $T_{ACL}=320$2 and $T_{ACL}=320$3; feed-forward MLPs with two linear layers and GELU; residual add; and row-wise softmax. Inference-only dropout is elided (Xu et al., 1 Jun 2026).

The principal transformer equations implemented in that work are:

$T_{ACL}=320$4

$T_{ACL}=320$5

$T_{ACL}=320$6

and

$T_{ACL}=320$7

The same study provides a compact complexity model. With sequence length $T_{ACL}=320$8, hidden size $T_{ACL}=320$9, heads $$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$0, and head dimension $$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$1, the attention block has rule-of-thumb FLOP count

$$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$2

while the FFN contributes

$$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$3

For BERT-base, with $$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$4 and $$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$5, the per-layer FFN dominates attention when $$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$6 is small. This complexity decomposition is central to the later scheduling rules: GEMM-heavy FFN and projection stages are compute-intensive, whereas reshape, transpose, masking, softmax, and normalization phases are comparatively memory-bound (Xu et al., 1 Jun 2026).

A significant conceptual shift follows from this extension. Earlier ACL usage relied on composing CNN-style primitives into inference graphs. The transformer work instead adds domain-specific kernels that encode the structure of attention blocks directly, reducing the need to emulate transformer behavior through generic operators.

4. Kernel organization, data layout, and precision modes

The transformer extension introduces both NE (CPU) and CL (GPU) kernels. Among the named components are NE/CL embedding lookup kernels; NEGEMM and CLGEMM for projections and FFN; fused NEGEMMBiasGELU and CLGEMMBiasGELU; BatchedQKMatMulScaleMask; NESoftmax and CLSoftmax; BatchedAVMatMul; ConcatHeads; NELayerNormalization and CLLayerNormalization; NEResidualAdd and CLResidualAdd; and NEGELU and CLGELU (Xu et al., 1 Jun 2026).

Sequences are represented as 2D tensors with shape $$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$7 in row-major form, mapped to ACL tensor dimensions as width $$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$8, height $$S=\frac{T_{TF}}{T_{ACL}}=\frac{420}{320}=1.3125\times$$9, channels $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$0. For multi-head operations, the data are reshaped to $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$1 and permuted between $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$2 and $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$3 to maximize contiguous access along $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$4. On the CPU side, NEON blocking interleaves panels in GEMM and uses FMA-heavy inner loops with register blocking such as $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$5 or $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$6, depending on $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$7, cache pressure, and $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$8. On Mali GPUs, CLGEMM uses tiled matmul with $$\frac{T_{TF}-T_{ACL}}{T_{TF}}\times 100\%=\frac{420-320}{420}\times 100\%=23.81\%.$$9-blocking; for $2.38$0 in $2.38$1, a typical choice is $2.38$2, $2.38$3, $2.38$4, adapted based on $2.38$5 to keep tiles in local memory below $2.38$6 KB. Padding strides to multiples of $2.38$7 floats is used to avoid bank conflicts (Xu et al., 1 Jun 2026).

Precision support is heterogeneous. FP32 is the default and is stated to ensure numerical equivalence to reference implementations. FP16/BF16 are supported where ACL offers half or bfloat types; FP16 reduces memory footprint and improves GPU throughput, while BF16 is recommended on CPU NEON where available to preserve dynamic range. INT8 (QASYMM8) and INT16 (QSYMM16) are supported through ACL quantized GEMM paths for FFN and linear layers, with per-tensor and per-channel schemes depending on the layer and offline calibration by KL-divergence or min/max. The recommended transformer deployment mode on edge devices is mixed precision: INT8 for linear/FFN and FP16/FP32 for attention softmax and layer normalization (Xu et al., 1 Jun 2026).

Memory management is treated as a first-class optimization target. The implementation pre-allocates tensors for all layer outputs, reuses buffers when topologically safe, and exploits shared host memory between CPU and GPU through OpenCL shared buffers. At CPU↔GPU boundaries, “shared” tensors are allocated in an OpenCL buffer backed by host-visible memory, enabling zero-copy hand-off and eliminating explicit memcpy between graph partitions. Consecutive same-device layers use single-target tensors to minimize runtime checks and mapping overhead (Xu et al., 1 Jun 2026).

5. Cooperative CPU–GPU execution on ARM-based HMPSoCs

The reference platform for the transformer study is the Khadas VIM3 BASIC with an Amlogic A331D SoC fabricated at 12 nm, combining a big.LITTLE CPU and a Mali G52 GPU. The high-performance runs used the quad Cortex-A73 big cluster at 2.2 GHz; the dual Cortex-A53 little cluster at 1.8 GHz was present but not used. The ISA was ARMv8-A with 128-bit NEON SIMD. The GPU was a quad-core Mali G52 of the Bifrost family at approximately 0.8 GHz. The software stack comprised Ubuntu 22.04.4, Linux kernel v4.9, ARM-CL v21.02 extended in the authors’ fork, and OpenCL 3.0. An empirical platform characteristic highlighted in the study is the asymmetry between CPU and GPU cache sizes: CPU L2 cache of 2 MB versus GPU L2 cache of approximately 128 KB (Xu et al., 1 Jun 2026).

The scheduler is formulated around arithmetic intensity,

$2.38$8

with memory-intensive operators defined as $2.38$9 FLOPs/byte and compute-intensive operators as $3.13$0 FLOPs/byte. On the reported platform, the decision rule is: if $3.13$1, assign all layers to CPU; otherwise map Embedding, SDPA softmax + masking + reshape + concat, and Add+LayerNorm to CPU, while mapping $3.13$2 projections, FFN matmuls, and the output projection $3.13$3 to GPU. The stated rationale is that memory-intensive operations benefit from CPU cache behavior and low-overhead scalar/vector control, while highly parallelizable GEMMs benefit from the GPU (Xu et al., 1 Jun 2026).

The execution pipeline is explicitly cooperative. CPU prepares embeddings and SDPA reshape while GPU performs $3.13$4 GEMMs; shared buffers carry intermediate tensors without memcpy; the host avoids blocking except at hand-off points; OpenCL events establish dependencies; and CPU-side NE kernels run on a thread pool. The implementation assembles blocks programmatically rather than through ACL’s Graph API in order to retain fine-grain control over placement and zero-copy buffers. The authors state that if the Graph API is used, explicit subgraph partitioning and replacement of Sender/Receiver with shared CLTensor buffers from their fork are recommended (Xu et al., 1 Jun 2026).

Thermal and DVFS effects are treated as operational constraints rather than afterthoughts. The study reports variability from thermal throttling and frequency scaling on embedded SoCs, and therefore recommends pinning big cores for NE workloads, avoiding GPU saturation when $3.13$5 is large, keeping GPU kernels short and tiled, and using cooperative scheduling to reduce peak power and improve sustained throughput (Xu et al., 1 Jun 2026).

6. Empirical performance, numerical behavior, and robustness

The transformer evaluation covered BERT-base, DistilBERT, MobileBERT, SqueezeBERT, and GPT-2 small. Unless noted otherwise, reported results use $3.13$6, batch size $3.13$7, default FP32, and causal masks for GPT-2. Baselines were TVM v0.18 on CPU and GPU and ExecuTorch on CPU; ExecuTorch lacked Mali GPU support. The comparison controlled for equivalent models, matched weight formats, and disabled training-only operations such as dropout (Xu et al., 1 Jun 2026).

The headline results are substantial. For single-processor inference, the extended ARM-CL achieved a $3.13$8 average speedup versus TVM on CPU across models and a $3.13$9 average speedup versus TVM on GPU. Cooperative CPU-GPU execution reduced latency by up to $75\%$0 relative to the best single-processor ARM-CL execution and was reported to win across all tested transformers. Representative end-to-end latencies, in milliseconds at $75\%$1 and batch $75\%$2, were: BERT-base $75\%$3 on CPU, $75\%$4 on GPU, and $75\%$5 cooperatively; DistilBERT $75\%$6, $75\%$7, and $75\%$8; MobileBERT $75\%$9, $90\%$0, and $90\%$1; SqueezeBERT $90\%$2, $90\%$3, and $90\%$4; and GPT-2 small $90\%$5, $90\%$6, and $90\%$7 (Xu et al., 1 Jun 2026).

Layer-level profiling for BERT-base at $90\%$8 clarifies the source of these gains. On CPU, Embedding was $90\%$9 ms, Attention Linear $9$0 ms, SDPA $9$1 ms, FF $9$2 ms, and Add+Norm $9$3 ms. On GPU, Embedding rose to $9$4 ms, Attention Linear dropped to $9$5 ms, SDPA was $9$6 ms, FF fell to $9$7 ms, and Add+Norm was $9$8 ms. The study therefore attributes the improvement to mapping FF and Attention Linear to GPU while keeping memory-bound Embedding, SDPA reshape/softmax, and Add+Norm on CPU (Xu et al., 1 Jun 2026).

The numerical validation is correspondingly explicit. Outputs were checked against reference implementations in PyTorch and TVM; FP32 paths matched within machine precision, while FP16 showed the small deviations expected for reduced precision. Stability measures included log-sum-exp stabilization for softmax row reductions, attention scaling by $9$9, and layer norm epsilon values such as $10$0. Causal masks for GPT-2 were implemented by setting upper-triangular positions to $10$1 before softmax, and padding masks were handled in BatchedQKMatMulScaleMask. Edge cases addressed in the paper include long sequences up to $10$2, variable sequence lengths through max-$10$3 pre-allocation and slicing, and support for both causal and bidirectional attention (Xu et al., 1 Jun 2026).

7. Limitations, portability, and research trajectory

The principal limitations reported for ACL-based transformer inference are memory capacity, GPU cache size, and quantization complexity. On typical embedded DRAM capacities of $10$4–$10$5 GB, BERT-base with approximately $10$6 MB of weights fit on the evaluated board, whereas BERT-large with approximately $10$7 GB of weights exceeded practical limits and reportedly crashed on the VIM3 setup. The Mali G52’s approximately $10$8 KB L2 cache limited gains when working sets exceeded local cache, and the study therefore recommends cooperative or CPU-only execution for $10$9 (Xu et al., 1 Jun 2026).

Quantization is presented as useful but nontrivial. The transformer paper supports INT8 and INT16 GEMM paths and recommends mixed precision, explicitly retaining attention softmax and layer normalization in FP16/FP32 for numerical stability. This is consistent with the earlier CNN case study, which showed that partial quantization inside TensorFlow made convolution kernels approximately $T_{ACL}=320$00 faster but increased end-to-end inference time by more than $T_{ACL}=320$01 ms because re-quantize/de-quantize overhead dominated the gain. The shared lesson is that quantization must be designed end to end rather than inserted piecemeal (Sun et al., 2017, Xu et al., 1 Jun 2026).

Portability is described with caution. The 2026 scheduling heuristics target Cortex-A73 plus Mali G52 and are said to generalize to similar big cores and Bifrost GPUs with shared host memory, but boards with larger GPU L2 caches may preserve GPU-only advantages at larger sequence lengths. The proposed future directions are kernel fusion beyond GEMM+Bias+GELU, FlashAttention-style implementations for reduced memory traffic at long $T_{ACL}=320$02, SVE2 exploitation on newer cores, Vulkan/Compute frontends where OpenCL is limited, and graph-level scheduling and autotuning integrated with the ACL Graph API. The transformer extensions and examples were released as open source under the MIT license at https://github.com/dondavan/fast-transformer-on-acl (Xu et al., 1 Jun 2026).

Within the available literature, ARM Compute Library therefore emerges as a kernel-centric inference substrate whose effectiveness depends on close alignment between model structure, operator coverage, and platform-aware scheduling. Its early documented strength lay in lean CNN runtimes on constrained ARM CPUs; its later development extends that role to transformer inference on ARM-based HMPSoCs through native attention kernels, zero-copy CPU–GPU cooperation, and explicit management of arithmetic intensity, cache behavior, and runtime overhead (Sun et al., 2017, Xu et al., 1 Jun 2026).