Protean Compiler for Adaptive LLVM Optimization

• Protean Compiler is an agile framework that integrates phase-ordering into LLVM via a native optimization level, addressing dependency challenges among passes.
• It employs simulated annealing over clustered LLVM pass subsequences with a learned cost model, delivering up to 15.7% speedup on select CBench applications.
• Built on a detailed 141-feature static analysis and a Transformer-based IR2Score, it supports fine-grained optimizations at module, function, and loop levels.

Protean Compiler is an agile framework that enables LLVM with built-in phase-ordering capabilities at a fine-grained scope. It is presented as a seamlessly integrated, full-fledged compiler extension rather than an external autotuning harness: it adds a new optimization level -OP, a new optimization phase ProteanOpt in the Clang driver, a new optimization tool protean, a complete library of more than 140 handcrafted static feature collection methods, and a search loop that combines clustered LLVM pass subsequences with a learned cost model. In the reported evaluation, the framework delivers speedup gains of up to 4.1% on average and up to 15.7% on select CBench applications with respect to LLVM’s -O3, while also supporting two-step optimization flows with other ML frameworks and LLMs (Ashouri et al., 5 Feb 2026).

1. Problem setting and design objective

Protean Compiler addresses the phase ordering problem, a long-standing compiler problem in which the effectiveness of one optimization pass depends on the presence, absence, and position of other passes. The broader autotuning literature treats optimization selection and phase ordering as the two central degrees of freedom in adaptive compilation, with search spaces that can grow as $n!$ or as $\sum_{i=0}^{l} n^i$, depending on whether repetition and variable sequence length are allowed (Ashouri et al., 2018).

Within that setting, Protean Compiler is defined by a specific systems claim: machine-learning-based approaches for compiler optimization had not been baked into the compiler seamlessly and had not materialized to be leveraged at a fine-grained scope of code segments. The framework therefore repositions phase ordering as an in-compiler capability. Rather than exposing an external search environment, it introduces a native optimization level that can optimize partitions of a program at module, call graph, function, or loop scope.

The design goal is not exhaustive search over individual LLVM passes. Instead, Protean Compiler constrains the search through clustered subsequences derived from the -O3 pipeline and evaluates candidate recipes with a learned static performance predictor. This suggests a deliberate balance between search tractability, LLVM integration, and compile-time overhead.

2. LLVM integration and system architecture

Protean Compiler is integrated into LLVM at the driver, tool, and pass levels (Ashouri et al., 5 Feb 2026). In the Clang driver, it defines a new phase ProteanOpt, inserted between phases::Compile and phases::Backend, and exposes a new command-line flag, -OP. If -OP is present, the Protean phase is executed; otherwise, the normal pipeline is used.

The execution flow has four stages. First, the front end takes C/C++ source and an optimization objective and produces LLVM IR. Second, the IR is partitioned into sub-programs or partitions; the simplest form is per-module partitioning, and the experimental evaluation primarily uses module-level scope. Third, an agile optimization loop operates per partition. For each partition, a simulated annealing engine generates candidate pass recipes, invokes a subprocess of the protean tool to apply the recipe to the partition’s IR, runs the IR2Score model on the resulting IR, and uses the predicted score to accept or reject the candidate. Fourth, once all partitions have been optimized, standard LLVM/Clang linking is performed. If LTO is enabled, link-time optimizations run as usual; Protean focuses on IR-level opt passes.

The protean executable is a modified version of opt with the same dependencies, but it replaces the fixed -O3 pipeline with the agile optimization loop. Candidate recipes are applied in subprocesses, which means that abnormal pass failures can be discarded without terminating the overall search. The framework also exposes configuration knobs for cooling schedule, maximum iterations, random seed, maximum temperature, crossover and mutation settings for a potential genetic-search variant, and two cost-function modes: IRAnalysis, which determines cost based on IR2Score, and MCA, which determines cost based on llvm-mca.

3. Pass clustering, recipe space, and simulated annealing

Protean Compiler does not search directly over LLVM’s full pass inventory. Instead, it clusters the -O3 pass list into five subsequences, labeled A through E, and searches over compositions of these subsequences (Ashouri et al., 5 Feb 2026). A recipe is therefore a string over {A,B,C,D,E}, such as ACDCD, CD, or ABBBA.

The subsequences preserve local pass orderings already designed by LLVM developers. Subsequence A includes globalopt, a cgscc pipeline with devirt, several inline forms, function-attrs, argpromotion, sroa, speculative execution, LICM, loop unswitching, loop unroll, early-cse, jump-threading, correlated-propagation, simplifycfg, instcombine, and aggressive-instcombine. Subsequence C includes sroa, gvn-hoist, mldst-motion, gvn, sccp, bdce, instcombine, jump-threading, adce, and memcpyopt. Subsequence E is loop-centric and includes loop-simplify, lcssa, loop-rotate, loop distribute, unroll-and-jam, loop vectorize, vector-combine, LICM, and loop-sink.

If the number of subsequences is $n=5$ and the maximum recipe length is $m$, the search space is

$\sum_{i=0}^{m} n^i.$

The reported tuning study uses automotive_susan_c to select a practical bound on recipe length. With $m=3$, the space contains 156 possible recipes and achieves relative best performance of approximately 0.85; with $m=4$, 781 recipes and approximately 0.92; with $m=5$, about 4K recipes and approximately 0.97; with $m=6$, 19K recipes and approximately 0.99; and with $m=7$, 97K recipes and 1.0. The chosen maximum length is 5, characterized as a good trade-off between tractability and observed performance.

Search proceeds via simulated annealing. For a current recipe with cost $\sum_{i=0}^{l} n^i$0, a candidate neighbor with cost $\sum_{i=0}^{l} n^i$1, and temperature $\sum_{i=0}^{l} n^i$2, the acceptance rule is

$\sum_{i=0}^{l} n^i$3

In the reported interpretation, if the predicted cost improves, the move is always accepted; if it worsens, it is accepted with an annealing probability. The default cooling schedule is geometric, although a linear schedule is also supported. The framework also includes early convergence logic: if the IR stops changing meaningfully, optimization for that partition can terminate early.

4. Static feature collection, IR2Score, and external ML interfaces

The static characterization layer of Protean Compiler is the Protean Feature Set, a library of 141 handcrafted static features implemented as an LLVM module pass (Ashouri et al., 5 Feb 2026). The feature space is explicitly multi-scope.

Module/global features number 10 and include FunctionCount, AverageBBPerFunction, TotalBBCount, TotalInstCount, TotalFunctionCalls, and GlobalVariableCount. Function-level features number 71 and cover metrics such as AverageComponentSize, SCCSize, IsLocal, CriticalEdgeCount, TotalEdgeCount, LoopCount, MedianCallsPerFunction, AverageCallsPerFunction, AverageStoreInstructionsPerFunction, AverageLoadInstructionsPerFunction, and AverageInstructionsPerFunction. Callee/caller features number 39 and include NumCallsiteInLoop, NestedInlines, SROALosses, IndirectCallPenalty, CallSiteCost, CalleeInstrPerLoop, CallerInstrPerLoop, CalleeAvgNestedLoopLevel, CallerAvgNestedLoopLevel, CalleeAvgVecInstr, CallerAvgVecInstr, and block- and frequency-oriented callsite features. Loop features number 31 and include LoopSize, NumLoadInstPerLoopNest, MaxLoopHeight, StepValueInt, FinalIVValueInt, InitialIVValueInt, TotLoopInstCount, TotLoopNestInstCount, IndVarSetSize, MaxTripCount, and TripCount.

PFS is registered at the end of each iteration’s pipeline, so the features describe the current post-optimization IR. Its output is scope-aware: module-level rows and function-plus-loop rows share a common schema, with non-applicable features zeroed. Candidate IRs are then scored by IR2Score, a Transformer-based model that uses sequences of feature snapshots corresponding to prefixes of applied pass sequences. The motivating representation is

$\sum_{i=0}^{l} n^i$4

The model choice is empirical: compared to MLP, CNN, and LSTM alternatives, the Transformer architecture produced higher accuracy and lower validation loss. Protean can consume either PFS features or external embeddings such as IR2VEC. Pearson-correlation analysis shows that IR2VEC has higher average correlation across its 300 dimensions, while PFS exhibits about a dozen strongly correlated features; in the reported experiments, the PFS-based IR2Score outperforms the IR2VEC-based version, plausibly because IR2VEC had been trained on SPEC rather than CBench.

IR2Score is deployed inside LLVM 19 through MLGO-style ModelRunner interfaces, with Ahead-Of-Time compiled models linked directly into the compiler. Protean also supports two-step workflows in which phase ordering is followed by pass-parameter optimization from external systems such as ACPO, or in which LLM-based source-to-source rewriting precedes Protean’s IR-level search. This IR-centric organization is consistent with broader work arguing that large, production-grade IR corpora are an enabling substrate for machine-learned compiler components (Grossman et al., 2023).

5. Experimental results and performance profile

The evaluation uses CBench on a Huawei Kunpeng 920 server, an aarch64 ARMv8.2 platform at 2.6 GHz running Linux, with LLVM 19 integrated with Protean Compiler (Ashouri et al., 5 Feb 2026). Workloads are pinned with numactl, runtimes are measured with Linux perf, the page cache is flushed before each run, and each benchmark is run five times. If variance exceeds 1%, the experiment is repeated. The reported mean variances are 0.58% for -O3, 0.71% for the IR2VEC-based IR2Score variant, and 0.49% for the PFS-based variant.

The reported geo-mean speedups over -O3 are:

Iterations	IR2VEC speedup	PFS speedup
10	1.022	1.025
50	1.033	1.035
100	1.035	1.039
500	1.037	1.041

At 500 iterations, the PFS configuration reaches approximately 4.1% average speedup with respect to -O3. Select benchmarks show much larger improvements, including 1.078× on consumer_jpeg_d, 1.139× on network_dijkstra, 1.157× on security_pgp_d, 1.065× on security_sha, and 1.050× on telecom_adpcm_d. The paper characterizes the maximum observed gain on select CBench applications as 15.7%.

Build-time overhead grows with iteration count, but the two model variants differ substantially:

Iterations	IR2VEC overhead	PFS overhead
10	30.918 s	31.531 s
50	132.467 s	129.270 s
100	219.830 s	240.693 s
500	732.556 s	320.446 s

The lower 500-iteration overhead for PFS is attributed to earlier convergence: IR2Score+PFS averages approximately 145 iterations per module before early stop, while the IR2VEC variant averages approximately 310. This suggests that the predictive quality of the cost model affects not only final code quality but also the practical compile-time envelope of the search.

Two-step optimization results are also reported. On consumer_jpeg_c, Protean alone, using IR2VEC and 20 iterations, finds recipe ACDCD and achieves 2.5% speedup over -O3. When combined with ACPO for inlining decisions inside subsequence A, the final runtime becomes 5.38 s, corresponding to approximately 8.5% speedup over -O3, with a 12% code-size increase relative to -O3 and 10% relative to Protean alone. On automotive_susan_c, an LLM, Qwen2.5-Coder-32B-Instruct, rewrites susan_corners() with loop unrolling factor 5 and yields 3.1% speedup over -O3; compiling the LLM-optimized source with Protean then yields a further 7% speedup over the LLM baseline, for an approximately 10.1% combined speedup with respect to -O3.

6. Relation to prior work, misconceptions, and limitations

Protean Compiler is positioned against prior ML-guided compiler optimization systems by the form of its integration rather than by the mere use of learning (Ashouri et al., 5 Feb 2026). MiCOMP is the closest comparison in the paper: it builds subsequences from LLVM -O3 and performs ML-guided iterative compilation, but it is implemented as a Python wrapper around LLVM 3.8, uses dynamic features from the MICA Pin tool, and operates at program-wide scope only. Protean instead is fully integrated into LLVM 19, uses static features, supports fine-grain scopes, and exposes a native -OP optimization level. Relative to MiCOMP, the reported extra performance is approximately 1–2%, depending on iteration count.

A common misconception would be to treat Protean as a pass-by-pass exhaustive search engine. It is not. The search space is deliberately constrained to recipes over five clustered subsequences, and the reported experiments primarily optimize at module level. Another misconception would be to read the framework as a dynamic or profile-driven optimizer. The deployed IR2Score path is static: it predicts performance from IR features rather than from hardware counters or repeated runtime execution.

Several limitations are explicit. Evaluation is limited to CBench on a single ARMv8.2 platform. Compile-time overhead can be substantial for large benchmarks and large iteration budgets. The framework supports module, call graph, function, and loop scope architecturally, but the published experiments focus on modules. Code-size growth can accompany higher performance, especially when Protean is composed with aggressive inlining systems such as ACPO. The authors also state that the project is intended for open-source release in the near future, so the implementation is presented as production-oriented but not yet publicly released.

Within those limits, Protean Compiler occupies a specific place in compiler research: it transforms phase ordering from an external autotuning experiment into a first-class optimization mode inside Clang/LLVM. A plausible implication is that its main significance lies less in any single heuristic than in the compiler architecture it proposes—clustered subsequence search, native ML inference, scope-aware static features, subprocess-isolated exploration, and explicit composition with third-party ML frameworks and LLMs.