nncase: Unified LLM Compilation Framework

• nncase is an open-source, end-to-end compilation framework that optimizes LLM deployment across heterogeneous storage architectures using a unified five-phase pipeline.
• It integrates e-graph-based optimization, parallel strategy search, kernel scheduling, and buffer-aware code generation to streamline data layout, distribution, and compute efficiency.
• Empirical evaluations demonstrate that nncase achieves competitive throughput against hand-tuned solutions like llama.cpp while outperforming Intel IPEX and MLC-LLM under varied memory hierarchies.

nncase is an open-source, end-to-end compilation framework for efficient LLM deployment on heterogeneous storage architectures. It is designed around the goal of “Compile once, adapt everywhere” for both uniform and non-uniform memory architectures, and it unifies optimization across diverse targets through a five-phase pipeline that combines e-graph-based optimization, parallel-strategy search, kernel scheduling, and buffer-aware code generation (Guo et al., 25 Dec 2025). In the formulation of the paper, nncase addresses the fragmented workflows and high adaptation costs that hinder traditional compilers under memory architecture heterogeneity, and it treats hardware hierarchy—SRAM/L1, caches, DRAM/HBM, and multi-chip networks—as a multi-level NUMA topology (Guo et al., 25 Dec 2025).

1. System model and compilation pipeline

The architectural premise of nncase is a NUMA abstraction that models any hardware hierarchy as a multi-level NUMA topology. Within that abstraction, optimization is guided by a Roofline-Driven Cost Model that provides a unified trade-off between compute throughput and data-movement cost (Guo et al., 25 Dec 2025). This design places memory movement and compute selection within a single optimization framework rather than treating them as separate backend concerns.

The compiler is described as a five-phase pipeline. First, it ingests a high-level graph into an e-graph. Second, it performs E-Graph-Based Optimization, driving both data-layout optimization through Auto Vectorize and distributed strategy search through Auto Distribution. Third, it applies Auto Scheduling for single-core kernels via structural and parametric search. Fourth, it performs Buffer-Aware Code Generation, including memory planning and NTT micro-kernels. Fifth, it links a final executable with high-performance µ-kernels (Guo et al., 25 Dec 2025).

The paper’s overall framing implies that nncase is not only an operator-lowering system but also a compiler that attempts global coordination across computation, data layout, distribution, and storage placement. This suggests a shift from narrowly local IR rewrite pipelines toward a whole-program optimization model in which memory hierarchy is a first-class compilation object.

2. E-graph term rewriting and global extraction

The central optimization substrate in nncase is an e-graph-based term rewriting engine. Its two basic data structures are an e-node, defined as an operator form referencing child e-classes, and an e-class, defined as a set of equivalent e-nodes (Guo et al., 25 Dec 2025). Equality Saturation is used to apply all rewrite rules non-destructively until saturation, so equivalent program variants are retained in parallel rather than being discarded by greedy rewriting.

The paper presents this mechanism as a way to alleviate the phase-ordering problem. Unlike greedy IR rewrites, e-graphs keep all variants in parallel, so there are no irreversible destructive updates (Guo et al., 25 Dec 2025). This matters because layout transformation, transpose motion, communication insertion, and kernel selection can interact strongly; preserving multiple equivalent forms allows later cost-based extraction to choose among them globally.

A representative transpose rewrite is given as:

$$\text{Binary}(T_{\text{perm}}(A), B) \to T_{\text{perm}}(\text{Binary}(A, T_{\text{perm}}^{-1}(B)))$$

under the rule CombineBinaryLeftTrans. The paper also lists CombineBinaryRightTrans, CombineUnaryTrans, FoldTwoTrans, and FoldNopTrans as key rewrite rules (Guo et al., 25 Dec 2025). These rules illustrate the compiler’s use of semantic equivalence classes to expose alternative placements of data transformation and computation.

After saturation, nncase performs extraction as WPMAXSAT. Each e-node receives a weight from a Roofline cost model based on memory traffic and compute cycles, and Weighted Partial MaxSAT is solved to pick one node per e-class minimizing total cost (Guo et al., 25 Dec 2025). This extraction procedure is significant because it converts a potentially large equivalence space into a single executable choice under a unified latency-oriented objective.

3. Layout adaptation and distributed strategy search

Auto Vectorize addresses conflicting layout requirements between vector units and tensor units, with the paper citing AVX-512 and AMX as an example (Guo et al., 25 Dec 2025). Its exploration rule, MetaPackOperation, is written as:

$$\text{Op}(\ldots) \to \text{Unpack}(\text{PackedOp}(\text{Pack}(\arg_1,\text{lanes}_1,\text{axes}_1),\ldots), \text{axes}^{-1})$$

and its pruning rule, FoldNopPack, is:

$\text{Pack}(\text{Unpack}(x,\ldots),\ldots) \to x$

(Guo et al., 25 Dec 2025). The workflow described for an Attention subgraph proceeds by generating both 1D vector-friendly and 2D blocked tensor-core variants in the e-graph, discovering a “pass-through” layout where Exp is fused over blocked tiles, and folding away redundant Pack/Unpack operations. Candidate layouts are then scored by the Roofline model, and extraction selects the end-to-end lowest-latency design (Guo et al., 25 Dec 2025).

Auto Distribution addresses parallel execution strategy. Its representation generalizes OneFlow’s SBP abstraction—Split, Broadcast, Partial-sum—to N-D SBP vectors:

$$\mathcal{P}_{nd} = [\,\text{sbp}_0,\dots,\text{sbp}_{N-1}\!]$$

(Guo et al., 25 Dec 2025). For an element-wise operation, the paper gives the validity constraint through the examples Valid: $A_{S(0)} + B_{S(0)} \to C_{S(0)}$ and Invalid: $A_{S(0)} + B_{S(1)} \to \bot$ (Guo et al., 25 Dec 2025). Search-space construction follows Algorithm 1: inputs generate all Box(input, sbp) e-nodes; compute nodes are formed by the Cartesian product of upstream e-classes plus optional Reshard(Box) nodes to yield all valid SBP signatures; and outputs are boxed back to host (Guo et al., 25 Dec 2025).

Communication cost is modeled per edge as $$\alpha(\text{message count}) + \beta(\text{message size})$$, combined with Roofline-based costs for compute edges (Guo et al., 25 Dec 2025). A memory constraint prunes any candidate whose per-device aggregate tensor sizes exceed capacity before extraction. In effect, nncase treats distribution choices as part of the same equivalence-and-extraction machinery used for algebraic and layout transformations. This suggests that communication planning is integrated into compiler optimization rather than delegated entirely to runtime systems.

4. Auto scheduling for single-core kernels

The Auto Schedule module uses tile-centric kernel decomposition to separate a Structural Part from a Parametric Part. The Structural Part covers loop fusion and loop orders, while the Parametric Part covers tile sizes and buffer placements (Guo et al., 25 Dec 2025). This decomposition allows structural exploration and parameter optimization to be coupled without collapsing them into a single monolithic search.

Structural search is performed via MCTS. The state is a Tiered Tile Graph described by the recursive notation

$Op^n=\{l_1^n,\dots\}(Op_a^{n-1},\dots)$

(Guo et al., 25 Dec 2025). The actions are merge(src, dst, level), which fuses two ops at a memory level, and reorder(i, n, loops), which permutes loops for tile $Op_i^n$ (Guo et al., 25 Dec 2025). Simulation uses a deterministic parametric solver based on MINLP to yield a latency reward.

Parametric optimization is then expressed through MINLP variables including

$$\mathsf{TileVar}_{l,op,i}\in\mathbb{Z}^+,\qquad \mathsf{Place}_{cl,op,b,e,sl}\in\{0,1\}$$

with static-analysis formulas for Extent, Buffer Size, Trip Count, and Data Traffic. The paper gives, for example,

$$\text{Op}(\ldots) \to \text{Unpack}(\text{PackedOp}(\text{Pack}(\arg_1,\text{lanes}_1,\text{axes}_1),\ldots), \text{axes}^{-1})$$0

and imposes constraints for domain coverage, unique buffer placement, fusion reuse, and memory capacity, including

$$\text{Op}(\ldots) \to \text{Unpack}(\text{PackedOp}(\text{Pack}(\arg_1,\text{lanes}_1,\text{axes}_1),\ldots), \text{axes}^{-1})$$1

The objective is

$$\text{Op}(\ldots) \to \text{Unpack}(\text{PackedOp}(\text{Pack}(\arg_1,\text{lanes}_1,\text{axes}_1),\ldots), \text{axes}^{-1})$$2

where

$$\text{Op}(\ldots) \to \text{Unpack}(\text{PackedOp}(\text{Pack}(\arg_1,\text{lanes}_1,\text{axes}_1),\ldots), \text{axes}^{-1})$$3

and $$\text{Op}(\ldots) \to \text{Unpack}(\text{PackedOp}(\text{Pack}(\arg_1,\text{lanes}_1,\text{axes}_1),\ldots), \text{axes}^{-1})$$4 is obtained from µ-kernel regression (Guo et al., 25 Dec 2025).

Within the paper’s framework, this scheduler is not an isolated autotuner; it is embedded after e-graph and distribution selection, so single-core kernel formation is conditioned on earlier global choices. A plausible implication is that the scheduler is intended to exploit cache locality only after data layout and device-level partitioning have already been globally optimized.

5. Buffer-aware code generation and the NTT/NTTD runtime substrate

The code generation phase is buffer-aware in two distinct senses: it explicitly reasons about aliasing and liveness for storage reuse, and it instantiates kernels against a specialized library substrate (Guo et al., 25 Dec 2025). Buffer scheduling begins with Bufferization plus Alias Analysis, under which view operations such as Reshape, Slice, and Squeeze share storage in a zero-copy manner. It then proceeds to Memory Planning through liveness-based bin-packing with a SAT solver for intermediates, while constants use SBP-driven placement (Guo et al., 25 Dec 2025).

The generated kernels target an NTT Library implemented with C++20 TMP and Concepts. The library uses a hybrid shape/stride model in which static dimensions are computed at compile time and dynamic dimensions at runtime. Its µ-kernels are register-level implementations exposing explicit vector types, with the paper giving ntt::vector<float,8> as an example, alongside pack/unpack intrinsics (Guo et al., 25 Dec 2025). This organization indicates that the backend is structured to preserve low-level control over vectorization and data packing decisions selected earlier in compilation.

The paper also introduces an NTTD Extension that encodes mesh topology and SBP in template parameters, with shard_policy::S\<2> given as an example, thereby eliminating runtime dispatch in distributed code (Guo et al., 25 Dec 2025). That encoding strategy ties the distribution model directly into code generation. This suggests that distributed execution structure is intended to be statically materialized rather than dynamically interpreted where possible.

The source code is available at https://github.com/kendryte/nncase (Guo et al., 25 Dec 2025).

6. Empirical results, comparisons, and stated implications

The reported evaluation uses an AMD Ryzen 9 5900X under single-core and multi-core settings with AVX2, batch=1, an 8-token prompt, and 100 runs (Guo et al., 25 Dec 2025). The comparisons are against mainstream frameworks including MLC LLM and Intel IPEX, with llama.cpp serving as the hand-optimized reference point.

For single-core throughput, the paper reports the following token/s figures (Guo et al., 25 Dec 2025):

Setting	Reported result	Comparative statement
Qwen3-0.6B F32	llama.cpp 10.61 > nncase 8.70 > IPEX 7.58	nncase (+15% vs IPEX) and ≫ MLC LLM
Qwen3-0.6B F16	llama.cpp 17.21 > nncase 13.87 > IPEX 10.22	nncase (+36% vs IPEX)
Qwen3-1.7B F16	llama.cpp 6.28 (≈) > nncase 5.09 > IPEX 4.20 > MLC 0.20	nncase (+21% vs IPEX)

For multi-core throughput, the paper reports (Guo et al., 25 Dec 2025):

Setting	Reported result	Comparative statement
Qwen3-0.6B F16 @4T	nncase 23.50 > llama.cpp 23.20 > IPEX 15.52	nncase leads
Qwen3-0.6B F16 @8T	nncase 23.98	bandwidth-bound
Qwen3-1.7B F16 @4T	nncase 8.85 > llama.cpp 8.34 > IPEX 6.93	nncase 74% scaling vs 32%

The paper’s stated takeaways are that nncase matches hand-tuned llama.cpp on multi-core thanks to static distributed partitioning, and that it holds a significant lead over Intel IPEX and MLC-LLM in both single- and multi-thread scenarios (Guo et al., 25 Dec 2025). It further concludes that a unified, e-graph-driven, cost-model-guided compilation framework can automatically resolve data layout, parallel strategy, and on-chip scheduling across heterogeneous memory and compute units; deliver performance on par with manual libraries such as llama.cpp and surpass other end-to-end compilers; provide a “compile once, adapt everywhere” workflow that lowers engineering effort for new hardware targets; and lay groundwork for future extensions including SIMT backends, deeper compute-communication overlap, and end-to-end compiler co-design across storage hierarchies (Guo et al., 25 Dec 2025).

In the paper’s own synthesis, nncase combines equality-saturation e-graphs, unified SBP search, MCTS+MINLP scheduling, and buffer-aware code generation to show that fully automated LLM compilation can achieve near-peak hardware performance on complex heterogeneous systems (Guo et al., 25 Dec 2025).