ANEForge: Direct Programming of Apple Neural Engine

• ANEForge is a Python package that directly programs the Apple Neural Engine, ensuring explicit ANE execution without silent CPU or GPU fallbacks.
• It compiles Python tensor graphs into single, fused ANE programs via Apple's private Espresso runtime, optimizing latency and efficiency.
• The tool supports both inference and on-engine training with advanced features like native fused attention, weight compression, and resident state management.

Searching arXiv for the specified paper to ground the article in the cited source. ANEForge is a Python package that programs the Apple Neural Engine (ANE), the fixed-function neural accelerator on every recent Apple device, directly and without CoreML. It exposes the ANE as a programmable accelerator for both practical model execution and hardware characterization, compiling a lazy tensor graph into one ANE program and dispatching that program through the same ANE daemon and kernel-driver stack as Apple’s internal framework. By fixing the dispatch unit to the ANE, ANEForge eliminates the silent CPU or GPU fallbacks that can occur under CoreML and presents the ANE as an explicit target rather than a scheduling option (Bryngelson, 12 Jun 2026).

1. Position within the Apple ANE software stack

The Apple Neural Engine is present in recent iPhones, iPads, and all Apple Silicon Macs. On M-series SoCs it is the lowest-power of the three programmable compute blocks—CPU, GPU, and ANE—for workloads that fit its model. In production, however, Apple exposes the ANE only via CoreML, and CoreML treats the ANE as one of several compute units. No CoreML configuration forces ANE execution, CPU is always a valid fallback, and there is no runtime API that tells which unit actually executed a given call. The practical consequence is that a model described as “ANE-accelerated” can silently run on CPU or GPU instead.

ANEForge addresses this opacity by entering Apple’s stack at the same point as CoreML’s internal runtime. It calls the private Espresso runtime C interface, e5rt, directly; this in turn interacts with the aned system daemon, the ANE kernel driver, and the ANE hardware. ANEForge never bypasses aned: all binaries are compiled and signed by Apple’s own tooling. The difference from standard CoreML workflows is therefore not a replacement of Apple’s lower-level execution path, but the removal of CoreML’s placement heuristics and the explicit fixation of execution on the ANE (Bryngelson, 12 Jun 2026).

2. Programming model and compilation pipeline

From the user’s perspective, ANEForge is a Python tensor library with an eager API but lazy semantics. Operator calls such as convolution, activation, and reduction do not execute immediately; they attach nodes to a lazy graph. Execution begins only when af.compile is invoked. That step lowers the graph into Model Intermediate Language (MIL), Apple’s internal IR used for ANE compilation, packs all constant weights into a single binary weight blob, invokes the ANE compiler via aned to produce a signed program binary, and returns a reusable Python callable wrapping the program handle.

The resulting dataflow is summarized in the paper as Python graph to MIL to program blob to e5rt dispatch to ANE silicon. A key design choice is that the full graph is compiled into one ANE program whenever possible, and each call to the compiled object is one dispatch to the ANE. Because the program handle is reused across calls, compile cost is amortized and repeated execution reduces to repeated dispatch of the same signed binary.

ANEForge targets Apple Silicon Macs running macOS 14 and later and requires only Python 3.10 and NumPy on the user side. The Objective-C++ shim that talks to Apple’s private frameworks is compiled locally on each machine and links against that system’s private libraries. This arrangement ties the package closely to specific OS and compiler versions, but it also preserves a direct path to the same runtime components used internally by Apple (Bryngelson, 12 Jun 2026).

3. Operator model, graph fusion, and native bridge access

ANEForge’s operator surface is organized around what the ANE compiler accepts and what the hardware exposes. The paper distinguishes two categories. The first is 58 fused operators, which can be lowered into a single ANE program. These include convolution and transposed convolution, dynamic convolution, matrix multiplication, batched matmul, einsum, a range of activations, several normalization families, pooling and upsampling, reductions and softmax, structural operators such as reshape and transpose, geometry and resampling operators, elementwise arithmetic and comparisons, and attention composites including scaled dot-product attention, multi-head attention, cross-attention, and GEGLU. When a graph is built from these operators, af.compile fuses the entire graph into a single ANE program.

The second category is 19 native bridge operators, corresponding to native hardware layer types that Apple’s public toolchain never emits but which are present in the ANE. To use them, ANEForge performs a graph cut and runs them as subprograms. Examples include native fused attention, argmax, topk, sort, point-cloud and geometry layers such as fps, radius_search, cross_product, cross_correlation, and cost_volume, along with layout transforms and operators such as dynamic_slice, input_view, and scaled_elementwise. Internally, these bridge operators are drawn from 26 native hardware layer types.

The package maintains a machine-checked capability registry for these layer types, and a conformance suite compiles and runs each one on the ANE to ensure continued availability across OS and compiler versions. Some operators, including sdpa, flatten, and local_response_norm, are route-selectable: they have both fused and bridge implementations. The optimizer af.tune can rewrite a bridge into the fused route if that yields a better program. This suggests that ANEForge is not merely a frontend to an existing compiler pipeline, but also a systematic instrument for mapping the boundary between compiler-supported fusion and directly exposed hardware functionality (Bryngelson, 12 Jun 2026).

4. Capabilities beyond standard inference

A central claim of ANEForge is that the ANE can be used not only for inference but also for training and numerical kernels, provided computation remains within the hardware’s fixed dataflow and FP16 precision constraints. One major capability is access to the ANE’s native fused attention layer through af.sdpa. This layer supports scaled dot-product attention, causal masking, single-query decode shape, and runtime additive masks. The paper notes that Apple’s public toolchain includes an SDPA operator but, as of the paper, always lowers it to individual matmuls and a softmax rather than emitting the native fused attention layer. ANEForge therefore exposes a hardware path that public tooling does not surface.

The package also supports streaming compressed weights from the ANE’s own dequantization path instead of pre-folding weights to FP16. The available modes are compress="int8", compress="int4", and compress="sparse". In the paper’s example of a 4096×4096 matmul, the weight blob is reported as 33.6 MB in FP16, approximately 8.4 MB in int8, and approximately 8.4 MB or less in int4 depending on packing, with int4 reported as approximately 4× versus FP16. Compression can be gated by compress_atol, so that layers whose compressed outputs deviate beyond a tolerance revert to coarser compression or FP16.

Another capability is state residency on the ANE via buffer aliasing. ANEForge uses this to keep decoder state, specifically key-value cache for autoregressive models, and optimizer state, such as Adam moments, resident across steps. In both cases the host supplies only new inputs each step while state never leaves the ANE. CoreML exposes key-value cache residency in a higher-level form via MLState in macOS 15 and later, but does not expose optimizer state residency.

ANEForge’s autograd module implements reverse-mode differentiation over its differentiable subset, allowing the forward pass, backward pass, and optimizer update to compile into ANE programs and run on the engine. The paper’s CIFAR-10 example reports 71.2% test accuracy, compared to 72.9% for a PyTorch implementation with the same topology, and states that gradients match PyTorch to cosine similarity 1.0 for key layers. The paper also notes an analytic gradient for softmax cross-entropy stabilized for FP16. At the same time, limitations are explicit: the dataplane is FP16, full Stable Diffusion sampling with classifier-free guidance does not close in half precision, and the ANE has no data-dependent control flow, so training loops and numerical methods must be expressible as static dataflow graphs with fixed iteration counts (Bryngelson, 12 Jun 2026).

5. Empirical validation and performance characteristics

ANEForge demonstrates end-to-end execution for several nontrivial models, each fused into one ANE program and validated against a framework reference. ResNet-18 uses 49 operators and is reported with cosine 1.0000 and exact top-1 match against torchvision. ViT-B/16 uses 299 operators and is reported with cosine 1.0000 and exact top-5 agreement. The MiniLM L6 sentence encoder is reported with cosine 1.0000 against a HuggingFace transformers reference. A Stable Diffusion U-Net forward pass, using 316 operators, is reported with relative error 0.042 against diffusers; only the forward pass is reported because the full diffusion sampling loop fails numerically in FP16. Every operator used in these models is part of the conformance suite, and the repository examples regenerate the reported metrics.

Latency measurements emphasize dispatch overhead and the importance of graph fusion. The ANE per-program dispatch floor is reported as approximately $70\,\mu\text{s}$, and a minimal fused program completes in approximately $90\,\mu\text{s}$. The paper writes this as

$$T_{\text{dispatch}} \approx 70\,\mu\text{s}, \qquad T_{\text{compute}} \approx 20\,\mu\text{s}$$

for a small program, so breaking a model into many small programs would cause the overhead term to dominate. This is the immediate rationale for compiling a large lazy graph into one program.

For a pretrained ResNet-18 forward pass, the paper reports ANE latency of approximately $0.33\,\text{ms}$ in one fused FP16 program, compared with approximately $2.0\,\text{ms}$ on GPU using torch-MPS in FP32 and approximately $6.0\,\text{ms}$ on CPU using PyTorch in FP32. The ANE rail power during the hot loop is reported as approximately $4.5\,\text{W}$, measured with powermetrics after subtracting idle power. The corresponding speedups are approximately $6.06\times$ versus GPU and approximately $18.18\times$ versus CPU. The paper also reports attention-specific optimization via af.tune: folding query, key, value, attention, and output projections into a single fused attention program yields 3.7×–5.3× speedups across tested sequence lengths, with overall attention in a Vision Transformer becoming 3.89× faster. On tested machines, the ANE appears as a single hardware lane, so performance gains come from maximizing work per program and using low-precision weight streaming rather than from concurrent dispatch (Bryngelson, 12 Jun 2026).

6. Constraints, compatibility, and research significance

ANEForge is deliberately constrained by the ANE’s fixed-function character. The dataplane is FP16, there is no data-dependent control flow, and the operator set is limited to operators accepted by the ANE compiler and passing the conformance gate. Unsupported operators or shapes are not silently emulated: the precompile validator may catch errors early, but ultimately af.compile fails if the MIL program cannot be compiled via aned. This rejection behavior is central to the package’s purpose, which is to characterize the ANE rather than provide transparent fallback execution.

The software stack is also constrained by private Apple APIs. Espresso, e5rt, and the ANE compiler are private and undocumented, so Apple offers no stability guarantees. For each release, ANEForge records the macOS version and ANECompiler version against which it has been verified. The paper currently reports verification on an M5 Pro and M1 Max under macOS 26.5 with ANECompiler 3520.4.1. Regression protection is provided by an operator corpus that compiles and runs every shipped operator on real hardware. Since hosted CI runners typically do not expose the ANE device, these tests run on physical Apple Silicon hardware. The paper also states that the work involves no Apple entitlement and no bypassing of code signing or SIP; all programs are compiled and signed by Apple’s own aned.

These design choices limit deployment. Because it links private frameworks, ANEForge is not suitable for App Store deployment and is aimed at research, local development, and on-device experiment and characterization on machines owned by the user. Within that scope, the package serves two roles simultaneously: a practical frontend for executing real models and a measurement instrument for determining what the ANE can compute, at what speed and power, and under full developer control. The demonstrations of on-engine training, resident state, fused attention, compressed weight streaming, and fixed-iteration numerical methods such as conjugate gradient and FFT suggest a broader research implication: neural accelerators exposed only as inference targets may admit a substantially richer programming model when the runtime stack is addressed directly (Bryngelson, 12 Jun 2026).